{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"02_PT_ML_Scripts.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"authorship_tag":"ABX9TyO1eTNOh6Czj2HkGEWrj4EI"},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"kTGdWhe0Uj1n","colab_type":"text"},"source":["# ML Scripts\n","\n","So far, we've done everything inside the Jupyter notebooks but we're going to now move our code into individual python scripts. We will lay out the code that needs to be inside each script but checkout the `API` lesson to see how it all comes together."]},{"cell_type":"markdown","metadata":{"id":"M5zF_oEsUj47","colab_type":"text"},"source":["<div align=\"left\">\n","<a href=\"https://github.com/madewithml/lessons/blob/master/notebooks/03_APIs/02_ML_Scripts/02_PT_ML_Scripts.ipynb\" role=\"button\"><img class=\"notebook-badge-image\" src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n","<a href=\"https://colab.research.google.com/github/madewithml/lessons/blob/master/notebooks/03_APIs/02_ML_Scripts/02_PT_ML_Scripts.ipynb\"><img class=\"notebook-badge-image\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"W15Q6XUX4TlA","colab_type":"text"},"source":["# data.py"]},{"cell_type":"markdown","metadata":{"id":"ZSNhxD2F5pfa","colab_type":"text"},"source":["## Load data"]},{"cell_type":"code","metadata":{"id":"SWaghwwR5FYv","colab_type":"code","colab":{}},"source":["import numpy as np\n","import pandas as pd\n","import random\n","import urllib"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"R3idv09r5La8","colab_type":"code","colab":{}},"source":["SEED = 1234\n","DATA_FILE = 'news.csv'\n","INPUT_FEATURE = 'title'\n","OUTPUT_FEATURE = 'category'"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"79NE_bWQSSzW","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","np.random.seed(SEED)\n","random.seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"jEbRJTz3VUZt","colab_type":"code","colab":{}},"source":["# Load data from GitHub to notebook's local drive\n","url = \"https://raw.githubusercontent.com/madewithml/lessons/master/data/news.csv\"\n","response = urllib.request.urlopen(url)\n","html = response.read()\n","with open(DATA_FILE, 'wb') as fp:\n","    fp.write(html)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7g-bnJrM5Qcz","colab_type":"code","outputId":"2ca19503-a8bd-44b1-bb91-89ed818bae25","executionInfo":{"status":"ok","timestamp":1584477909847,"user_tz":420,"elapsed":4700,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Load data\n","df = pd.read_csv(DATA_FILE, header=0)\n","X = df[INPUT_FEATURE].values\n","y = df[OUTPUT_FEATURE].values\n","df.head(5)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>title</th>\n","      <th>category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Oil prices soar to all-time record, posing new...</td>\n","      <td>Business</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                               title  category\n","0  Wall St. Bears Claw Back Into the Black (Reuters)  Business\n","1  Carlyle Looks Toward Commercial Aerospace (Reu...  Business\n","2    Oil and Economy Cloud Stocks' Outlook (Reuters)  Business\n","3  Iraq Halts Oil Exports from Main Southern Pipe...  Business\n","4  Oil prices soar to all-time record, posing new...  Business"]},"metadata":{"tags":[]},"execution_count":5}]},{"cell_type":"markdown","metadata":{"id":"2ipAPag59QRE","colab_type":"text"},"source":["## Preprocessing"]},{"cell_type":"code","metadata":{"id":"WpnTn7ZCw6wa","colab_type":"code","colab":{}},"source":["import re"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"WcFIjdL59hW8","colab_type":"code","colab":{}},"source":["LOWER = True\n","FILTERS = r\"[!\\\"'#$%&()*\\+,-./:;<=>?@\\\\\\[\\]^_`{|}~]\""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YFvFH3Rg9Pj3","colab_type":"code","colab":{}},"source":["def preprocess_texts(texts, lower, filters):\n","    preprocessed_texts = []\n","    for text in texts: \n","        if lower:\n","            text = ' '.join(word.lower() for word in text.split(\" \"))\n","        text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n","        text = re.sub(filters, r\"\", text)\n","        text = re.sub(' +', ' ', text) # remove multiple spaces\n","        text = text.strip()\n","        preprocessed_texts.append(text)\n","    return preprocessed_texts"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"CZ-VIAak9xWN","colab_type":"code","outputId":"2815d339-4638-41b4-a003-380faf9159b9","executionInfo":{"status":"ok","timestamp":1584477912007,"user_tz":420,"elapsed":6729,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["original_text = X[0]\n","X = np.array(preprocess_texts(X, lower=LOWER, filters=FILTERS))\n","print (f\"{original_text} → {X[0]}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Wall St. Bears Claw Back Into the Black (Reuters) → wall st bears claw back into the black reuters\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_6Ox-7HQ5q_C","colab_type":"text"},"source":["## Split data"]},{"cell_type":"code","metadata":{"id":"Md_QoxVd5m0j","colab_type":"code","colab":{}},"source":["import collections\n","from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9S4hM6k65s43","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"DPhUEpc05s7b","colab_type":"code","colab":{}},"source":["def train_val_test_split(X, y, val_size, test_size, shuffle):\n","    X_train, X_test, y_train, y_test = train_test_split(\n","        X, y, test_size=test_size, stratify=y, shuffle=shuffle)\n","    X_train, X_val, y_train, y_val = train_test_split(\n","        X_train, y_train, test_size=val_size, stratify=y_train, shuffle=shuffle)\n","    return X_train, X_val, X_test, y_train, y_val, y_test"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7Y5CV_wA5tAx","colab_type":"code","outputId":"edee48dc-8aac-4561-9abd-8fee9b2859ea","executionInfo":{"status":"ok","timestamp":1584477912345,"user_tz":420,"elapsed":6949,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Create data splits\n","X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n","    X=X, y=y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n","class_counts = dict(collections.Counter(y))\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_val: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")\n","print (f\"{X_train[0]} → {y_train[0]}\")\n","print (f\"Classes: {class_counts}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["X_train: (86700,), y_train: (86700,)\n","X_val: (15300,), y_val: (15300,)\n","X_test: (18000,), y_test: (18000,)\n","pga overhauls system for ryder cup points → Sports\n","Classes: {'Business': 30000, 'Sci/Tech': 30000, 'Sports': 30000, 'World': 30000}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"IG1t4tNc6F13","colab_type":"text"},"source":["# tokenizers.py"]},{"cell_type":"markdown","metadata":{"id":"zCrWtUL86JHa","colab_type":"text"},"source":["## Tokenizer"]},{"cell_type":"code","metadata":{"id":"S5wM4W4pBFTI","colab_type":"code","colab":{}},"source":["import json\n","import re"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"S0XD87Oq5wdu","colab_type":"code","colab":{}},"source":["SEPARATOR = ' ' # word level"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"6Wdznl966WEj","colab_type":"code","colab":{}},"source":["class Tokenizer(object):\n","    def __init__(self, separator, pad_token='<PAD>', oov_token='<UNK>',\n","                 token_to_index={'<PAD>': 0, '<UNK>': 1}):\n","        self.separator = separator\n","        self.oov_token = oov_token\n","        self.token_to_index = token_to_index\n","        self.index_to_token = {v: k for k, v in self.token_to_index.items()}\n","\n","    def __len__(self):\n","        return len(self.token_to_index)\n","    \n","    def __str__(self):\n","        return f\"<Tokenizer(num_tokens={len(self)})>\"\n","\n","    def fit_on_texts(self, texts):\n","        for text in texts:\n","            for token in text.split(self.separator):\n","                if token not in self.token_to_index:\n","                    index = len(self)\n","                    self.token_to_index[token] = index\n","                    self.index_to_token[index] = token\n","        return self\n","\n","    def texts_to_sequences(self, texts):\n","        sequences = []\n","        for text in texts:\n","            sequence = []\n","            for token in text.split(self.separator):\n","                sequence.append(self.token_to_index.get(\n","                    token, self.token_to_index[self.oov_token]))\n","            sequences.append(sequence)\n","        return sequences\n","            \n","    def sequences_to_texts(self, sequences):\n","        texts = []\n","        for sequence in sequences:\n","            text = []\n","            for index in sequence:\n","                text.append(self.index_to_token.get(index, self.oov_token))\n","            texts.append(self.separator.join([token for token in text]))\n","        return texts\n","\n","    def save(self, fp):\n","        with open(fp, 'w') as fp:\n","            contents = {\n","                'separator': self.separator,\n","                'oov_token': self.oov_token,\n","                'token_to_index': self.token_to_index\n","            }\n","            json.dump(contents, fp, indent=4, sort_keys=False)\n","\n","    @classmethod\n","    def load(cls, fp):\n","        with open(fp, 'r') as fp:\n","            kwargs = json.load(fp=fp)\n","        return cls(**kwargs)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"_hqeHgj-6WHk","colab_type":"code","outputId":"5f79b445-6a7c-460c-fea0-0649c107bd9f","executionInfo":{"status":"ok","timestamp":1584477912346,"user_tz":420,"elapsed":6859,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Input vectorizer\n","X_tokenizer = Tokenizer(separator=SEPARATOR)\n","X_tokenizer.fit_on_texts(texts=X_train)\n","vocab_size = len(X_tokenizer)\n","print (X_tokenizer)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<Tokenizer(num_tokens=35635)>\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"l3OQiMq3-kOC","colab_type":"code","outputId":"885bfd61-05b8-4542-b52e-4b292504b745","executionInfo":{"status":"ok","timestamp":1584477912347,"user_tz":420,"elapsed":6827,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Convert text to sequence of tokens\n","original_text = X_train[0]\n","X_train = np.array(X_tokenizer.texts_to_sequences(X_train))\n","X_val = np.array(X_tokenizer.texts_to_sequences(X_val))\n","X_test = np.array(X_tokenizer.texts_to_sequences(X_test))\n","preprocessed_text = X_tokenizer.sequences_to_texts([X_train[0]])\n","print (f\"{original_text} \\n\\t→ {preprocessed_text} \\n\\t→ {X_train[0]}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["pga overhauls system for ryder cup points \n","\t→ ['pga overhauls system for ryder cup points'] \n","\t→ [2, 3, 4, 5, 6, 7, 8]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"HjUyz1xTEr1S","colab_type":"code","colab":{}},"source":["# Save tokenizer\n","X_tokenizer.save(fp='X_tokenizer.json')"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ODKSbuHdEr4D","colab_type":"code","outputId":"9c4aad8e-9236-4619-854e-d5d23e02efea","executionInfo":{"status":"ok","timestamp":1584477912347,"user_tz":420,"elapsed":6750,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Load tokenizer\n","X_tokenizer = Tokenizer.load(fp='X_tokenizer.json')\n","print (X_tokenizer)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<Tokenizer(num_tokens=35635)>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ZWGeTWGx6SO-","colab_type":"text"},"source":["## Label Encoder"]},{"cell_type":"code","metadata":{"id":"zN4eiU3UDk7z","colab_type":"code","colab":{}},"source":["class LabelEncoder(object):\n","    def __init__(self, class_to_index={}):\n","        self.class_to_index = class_to_index\n","        self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n","        self.classes = list(self.class_to_index.keys())\n","\n","    def __len__(self):\n","        return len(self.class_to_index)\n","\n","    def __str__(self):\n","        return f\"<LabelEncoder(num_classes={len(self)})>\"\n","\n","    def fit(self, y_train):\n","        for i, class_ in enumerate(np.unique(y_train)):\n","            self.class_to_index[class_] = i\n","        self.index_to_class = {v: k for k, v in self.class_to_index.items()}\n","        self.classes = list(self.class_to_index.keys())\n","        return self\n","    \n","    def transform(self, y):\n","        return np.array([self.class_to_index[class_] for class_ in y])\n","\n","    def decode(self, index):\n","        return self.index_to_class.get(index, None)\n","    \n","    def save(self, fp):\n","        with open(fp, 'w') as fp:\n","            contents = {\n","                'class_to_index': self.class_to_index\n","            }\n","            json.dump(contents, fp, indent=4, sort_keys=False)\n","\n","    @classmethod\n","    def load(cls, fp):\n","        with open(fp, 'r') as fp:\n","            kwargs = json.load(fp=fp)\n","        return cls(**kwargs)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2L-7aZGiHt0d","colab_type":"code","colab":{}},"source":["# Output vectorizer\n","y_tokenizer = LabelEncoder()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2tL6uP3GMUTM","colab_type":"code","outputId":"e5ba7396-7141-47e6-c3ee-092e83a39cd8","executionInfo":{"status":"ok","timestamp":1584477912348,"user_tz":420,"elapsed":6678,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Fit on train data\n","y_tokenizer = y_tokenizer.fit(y_train)\n","print (y_tokenizer)\n","classes = y_tokenizer.classes\n","print (f\"classes: {classes}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<LabelEncoder(num_classes=4)>\n","classes: ['Business', 'Sci/Tech', 'Sports', 'World']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"-S5aVuoAMcFF","colab_type":"code","outputId":"25c1bc61-e630-4166-937c-8b0be8717210","executionInfo":{"status":"ok","timestamp":1584477912349,"user_tz":420,"elapsed":6638,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Convert labels to tokens\n","class_ = y_train[0]\n","y_train = y_tokenizer.transform(y_train)\n","y_val = y_tokenizer.transform(y_val)\n","y_test = y_tokenizer.transform(y_test)\n","print (f\"{class_} → {y_train[0]}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Sports → 2\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"tS5X_y5HMoqV","colab_type":"code","outputId":"b9aedf9f-664b-4714-e0dc-a7620d0ecc08","executionInfo":{"status":"ok","timestamp":1584477912349,"user_tz":420,"elapsed":6618,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Class weights\n","counts = np.bincount(y_train)\n","class_weights = {i: 1.0/count for i, count in enumerate(counts)}\n","print (f\"class counts: {counts},\\nclass weights: {class_weights}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["class counts: [21675 21675 21675 21675],\n","class weights: {0: 4.61361014994233e-05, 1: 4.61361014994233e-05, 2: 4.61361014994233e-05, 3: 4.61361014994233e-05}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"XwGVyyCYMwRc","colab_type":"code","colab":{}},"source":["# Save label encoder\n","y_tokenizer.save(fp='y_tokenizer.json')"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"jn7NWNDLM323","colab_type":"code","outputId":"546056f4-2bf4-40bb-b3dc-f51e6945756e","executionInfo":{"status":"ok","timestamp":1584477912350,"user_tz":420,"elapsed":6555,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Load label encoder\n","y_tokenizer = LabelEncoder.load(fp='y_tokenizer.json')\n","print (y_tokenizer)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<LabelEncoder(num_classes=4)>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"TslRxG8LNGw-","colab_type":"text"},"source":["# datasets.py"]},{"cell_type":"code","metadata":{"id":"hECwcmSoNAyE","colab_type":"code","colab":{}},"source":["import math\n","import torch\n","import torch.nn as nn\n","from torch.utils.data import Dataset\n","from torch.utils.data import DataLoader"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"DpVnkdR2R_z8","colab_type":"code","colab":{}},"source":["BATCH_SIZE = 128\n","FILTER_SIZES = [2, 3, 4]"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"L9YuYoy2SCcI","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","torch.manual_seed(SEED)\n","torch.cuda.manual_seed(SEED)\n","torch.cuda.manual_seed_all(SEED) # multi-GPU.\n","torch.backends.cudnn.benchmark = False\n","torch.backends.cudnn.deterministic = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Lbi8CYWuSDzM","colab_type":"code","outputId":"7631384b-745f-46e3-94be-ed8e7c524431","executionInfo":{"status":"ok","timestamp":1584480204230,"user_tz":420,"elapsed":884,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["USE_CUDA = True\n","DEVICE = torch.device('cuda' if (torch.cuda.is_available() and USE_CUDA) else 'cpu')\n","print (DEVICE)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["cuda\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CsK1RW9rVDte","colab_type":"text"},"source":["## Pad"]},{"cell_type":"code","metadata":{"id":"TYL2P-4XUYeX","colab_type":"code","colab":{}},"source":["def pad_sequences(X, max_seq_len):\n","    sequences = np.zeros((len(X), max_seq_len))\n","    for i, sequence in enumerate(X):\n","        sequences[i][:len(sequence)] = sequence\n","    return sequences "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"DmsnB81HUdkE","colab_type":"code","outputId":"15b5b44b-3601-4545-a03d-8c951e5a2efb","executionInfo":{"status":"ok","timestamp":1584477916491,"user_tz":420,"elapsed":10556,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Pad sequences\n","inputs = [[1,2,3], [1,2,3,4], [1,2]]\n","max_seq_len = max(len(x) for x in inputs)\n","padded_inputs = pad_sequences(X=inputs, max_seq_len=max_seq_len)\n","print (padded_inputs.shape)\n","print (padded_inputs)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["(3, 4)\n","[[1. 2. 3. 0.]\n"," [1. 2. 3. 4.]\n"," [1. 2. 0. 0.]]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Wm91J4kxVF6e","colab_type":"text"},"source":["## Dataset"]},{"cell_type":"code","metadata":{"id":"mi0BYzK7SFLm","colab_type":"code","colab":{}},"source":["class TextDataset(Dataset):\n","    def __init__(self, X, y, batch_size, max_filter_size):\n","        self.X = X\n","        self.y = y\n","        self.batch_size = batch_size\n","        self.max_filter_size = max_filter_size\n","\n","    def __len__(self):\n","        return len(self.y)\n","\n","    def __str__(self):\n","        return f\"<Dataset(N={len(self)}, batch_size={self.batch_size}, num_batches={self.get_num_batches()})>\"\n","\n","    def __getitem__(self, index):\n","        X = self.X[index]\n","        y = self.y[index]\n","        return X, y\n","\n","    def get_num_batches(self):\n","        return math.ceil(len(self)/self.batch_size)\n","\n","    def collate_fn(self, batch):\n","        \"\"\"Processing on a batch.\"\"\"\n","        # Get inputs\n","        X = np.array(batch)[:, 0]\n","        y = np.array(batch)[:, 1]\n","\n","        # Pad inputs\n","        max_seq_len = max(self.max_filter_size, max([len(x) for x in X]))\n","        X = pad_sequences(X=X, max_seq_len=max_seq_len)\n","\n","        return X, y\n","\n","    def generate_batches(self, shuffle=False, drop_last=False):\n","        dataloader = DataLoader(dataset=self, batch_size=self.batch_size, \n","                                collate_fn=self.collate_fn, shuffle=shuffle, \n","                                drop_last=drop_last, pin_memory=True)\n","        for (X, y) in dataloader:\n","            X = torch.LongTensor(X.astype(np.int32))\n","            y = torch.LongTensor(y.astype(np.int32))\n","            yield X, y"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"0Lg66kSISyfe","colab_type":"code","outputId":"1de7d4a4-9aa1-438d-a529-38c9a3d45583","executionInfo":{"status":"ok","timestamp":1584480170763,"user_tz":420,"elapsed":906,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Create datasets\n","train_set = TextDataset(X=X_train, y=y_train, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","val_set = TextDataset(X=X_val, y=y_val, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","test_set = TextDataset(X=X_test, y=y_test, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","print (train_set)\n","print (train_set[0])"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<Dataset(N=86700, batch_size=128, num_batches=678)>\n","([2, 3, 4, 5, 6, 7, 8], 2)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"PRPhTzc7TOq3","colab_type":"code","outputId":"e94f4ed8-3c93-4f78-e7bd-7e1a11b017ee","executionInfo":{"status":"ok","timestamp":1584480171015,"user_tz":420,"elapsed":308,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Generate batch\n","batch_X, batch_y = next(iter(test_set.generate_batches()))\n","print (batch_X.shape)\n","print (batch_y.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.Size([128, 13])\n","torch.Size([128])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"F_J-dalSjlaJ","colab_type":"text"},"source":["# utils.py"]},{"cell_type":"markdown","metadata":{"id":"sBEXKnUfjnp5","colab_type":"text"},"source":["## Embeddings"]},{"cell_type":"code","metadata":{"id":"UhHaWebnjrkw","colab_type":"code","colab":{}},"source":["from io import BytesIO\n","from urllib.request import urlopen\n","from zipfile import ZipFile"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"eRssFQL4j0-H","colab_type":"code","colab":{}},"source":["EMBEDDING_DIM = 100"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"cszOugYTkBOs","colab_type":"code","colab":{}},"source":["def load_glove_embeddings(embeddings_file):\n","    \"\"\"Load embeddings from a file.\"\"\"\n","    embeddings = {}\n","    with open(embeddings_file, \"r\") as fp:\n","        for index, line in enumerate(fp):\n","            values = line.split()\n","            word = values[0]\n","            embedding = np.asarray(values[1:], dtype='float32')\n","            embeddings[word] = embedding\n","    return embeddings"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"3zHy7mABkIPa","colab_type":"code","colab":{}},"source":["def make_embeddings_matrix(embeddings, token_to_index, embedding_dim):\n","    \"\"\"Create embeddings matrix to use in Embedding layer.\"\"\"\n","    embedding_matrix = np.zeros((len(token_to_index), embedding_dim))\n","    for word, i in token_to_index.items():\n","        embedding_vector = embeddings.get(word)\n","        if embedding_vector is not None:\n","            embedding_matrix[i] = embedding_vector\n","    return embedding_matrix"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Gj2_vT7kjora","colab_type":"code","outputId":"6d262d66-3e3f-45b4-fd4b-db887f584b0c","executionInfo":{"status":"ok","timestamp":1584478307127,"user_tz":420,"elapsed":400889,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Unzip the file (may take ~3-5 minutes)\n","resp = urlopen('http://nlp.stanford.edu/data/glove.6B.zip')\n","zipfile = ZipFile(BytesIO(resp.read()))\n","zipfile.namelist()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['glove.6B.50d.txt',\n"," 'glove.6B.100d.txt',\n"," 'glove.6B.200d.txt',\n"," 'glove.6B.300d.txt']"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"code","metadata":{"id":"_UZ4LB2gj3-Z","colab_type":"code","outputId":"d20d12c4-9353-4543-bde2-59ea3ecbc4fd","executionInfo":{"status":"ok","timestamp":1584478311445,"user_tz":420,"elapsed":405157,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Write embeddings to file\n","embeddings_file = 'glove.6B.{0}d.txt'.format(EMBEDDING_DIM)\n","zipfile.extract(embeddings_file)\n","!ls"],"execution_count":0,"outputs":[{"output_type":"stream","text":["glove.6B.100d.txt  news.csv  sample_data  X_tokenizer.json  y_tokenizer.json\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"TwIhrH3YkLA2","colab_type":"code","outputId":"b427fc4b-2439-4d83-e906-05a8b440d7db","executionInfo":{"status":"ok","timestamp":1584478323457,"user_tz":420,"elapsed":417108,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Create embeddings\n","embeddings_file = 'glove.6B.{0}d.txt'.format(EMBEDDING_DIM)\n","glove_embeddings = load_glove_embeddings(embeddings_file=embeddings_file)\n","embedding_matrix = make_embeddings_matrix(\n","    embeddings=glove_embeddings, token_to_index=X_tokenizer.token_to_index, \n","    embedding_dim=EMBEDDING_DIM)\n","print (embedding_matrix.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["(35635, 100)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"qBtecYquVBCW","colab_type":"text"},"source":["# model.py"]},{"cell_type":"markdown","metadata":{"id":"0MA-MQ89jwOS","colab_type":"text"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"CcHfQq9rUOuG","colab_type":"code","colab":{}},"source":["import torch.nn.functional as F"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"gN0ez7Y-jzsh","colab":{}},"source":["NUM_FILTERS = 50\n","HIDDEN_DIM = 128\n","DROPOUT_P = 0.1"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"j6yo6ZU4VMlP","colab_type":"code","colab":{}},"source":["class TextCNN(nn.Module):\n","    def __init__(self, embedding_dim, vocab_size, num_filters, filter_sizes, \n","                 hidden_dim, dropout_p, num_classes, pretrained_embeddings=None, \n","                 freeze_embeddings=False, padding_idx=0):\n","        super(TextCNN, self).__init__()\n","\n","        # Initialize embeddings\n","        if pretrained_embeddings is None:\n","            self.embeddings = nn.Embedding(\n","                embedding_dim=embedding_dim, num_embeddings=vocab_size, \n","                padding_idx=padding_idx)\n","        else:\n","            pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n","            self.embeddings = nn.Embedding(\n","                embedding_dim=embedding_dim, num_embeddings=vocab_size, \n","                padding_idx=padding_idx, _weight=pretrained_embeddings)\n","        \n","        # Freeze embeddings or not\n","        if freeze_embeddings:\n","            self.embeddings.weight.requires_grad = False\n","        \n","        # Conv weights\n","        self.filter_sizes = filter_sizes\n","        self.conv = nn.ModuleList(\n","            [nn.Conv1d(in_channels=embedding_dim, \n","                       out_channels=num_filters, \n","                       kernel_size=f) for f in filter_sizes])\n","     \n","        # FC weights\n","        self.dropout = nn.Dropout(dropout_p)\n","        self.fc1 = nn.Linear(num_filters*len(filter_sizes), hidden_dim)\n","        self.fc2 = nn.Linear(hidden_dim, num_classes)\n","\n","    def forward(self, x_in, channel_first=False):\n","        \n","        # Embed\n","        x_in = self.embeddings(x_in)\n","        if not channel_first:\n","            x_in = x_in.transpose(1, 2) # (N, channels, sequence length)\n","            \n","        # Conv + pool\n","        z = []\n","        conv_outputs = [] # for interpretability\n","        max_seq_len = x_in.shape[2]\n","        for i, f in enumerate(self.filter_sizes):\n","            # `SAME` padding\n","            padding_left = int((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2)\n","            padding_right = int(math.ceil((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2))\n","\n","            # Conv + pool\n","            _z = self.conv[i](F.pad(x_in, (padding_left, padding_right)))\n","            conv_outputs.append(_z)\n","            _z = F.max_pool1d(_z, _z.size(2)).squeeze(2)\n","            z.append(_z)\n","        \n","        # Concat conv outputs\n","        z = torch.cat(z, 1)\n","\n","        # FC layers\n","        z = self.fc1(z)\n","        z = self.dropout(z)\n","        logits = self.fc2(z)\n","\n","        return conv_outputs, logits"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ed42EprofmZx","colab_type":"code","outputId":"9ec4c75b-980a-453e-f4b5-6a8da9df4620","executionInfo":{"status":"ok","timestamp":1584480262017,"user_tz":420,"elapsed":2302,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Initialize model\n","model = TextCNN(embedding_dim=EMBEDDING_DIM,\n","                vocab_size=vocab_size,\n","                num_filters=NUM_FILTERS,\n","                filter_sizes=FILTER_SIZES,\n","                hidden_dim=HIDDEN_DIM,\n","                dropout_p=DROPOUT_P,\n","                num_classes=len(classes),\n","                pretrained_embeddings=embedding_matrix,\n","                freeze_embeddings=False).to(DEVICE)\n","print (model.named_parameters)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of TextCNN(\n","  (embeddings): Embedding(35635, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=128, bias=True)\n","  (fc2): Linear(in_features=128, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"HaxsbQUZgdeo","colab_type":"text"},"source":["# train.py"]},{"cell_type":"markdown","metadata":{"id":"vw-OKCRxjzin","colab_type":"text"},"source":["## Training"]},{"cell_type":"code","metadata":{"id":"PWrjBATegsbX","colab_type":"code","outputId":"448567b7-ccc5-485f-e5f4-6ba8b459eb99","executionInfo":{"status":"ok","timestamp":1584480262018,"user_tz":420,"elapsed":1862,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["from pathlib import Path\n","from torch.optim import Adam\n","from torch.optim.lr_scheduler import ReduceLROnPlateau\n","from torch.utils.tensorboard import SummaryWriter\n","%load_ext tensorboard"],"execution_count":0,"outputs":[{"output_type":"stream","text":["The tensorboard extension is already loaded. To reload it, use:\n","  %reload_ext tensorboard\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"V5JpQPF7itU7","colab_type":"code","colab":{}},"source":["LEARNING_RATE = 1e-4\n","PATIENCE = 3\n","NUM_EPOCHS = 100"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"G96Iwt6wqmHV","colab_type":"code","colab":{}},"source":["def train_step(model, device, dataset, optimizer):\n","    \"\"\"Train step.\"\"\"\n","    # Set model to train mode\n","    model.train()\n","    train_loss = 0.\n","    correct = 0\n","\n","    # Iterate over train batches\n","    for i, (X, y) in enumerate(dataset.generate_batches()):\n","\n","        # Set device\n","        X, y = X.to(device), y.to(device)\n","\n","        # Reset gradients\n","        optimizer.zero_grad()\n","\n","        # Forward pass\n","        _, logits = model(X)\n","\n","        # Define loss\n","        loss = F.cross_entropy(logits, y)\n","\n","        # Backward pass\n","        loss.backward()\n","\n","        # Update weights\n","        optimizer.step()\n","\n","        # Metrics\n","        y_pred = logits.max(dim=1)[1] \n","        correct += torch.eq(y_pred, y).sum().item()\n","        train_loss += (loss.item() - train_loss) / (i + 1)\n","\n","    train_acc = 100. * correct / len(dataset)\n","    return train_loss, train_acc"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"hJMliiu6vzf2","colab_type":"code","colab":{}},"source":["def test_step(model, device, dataset):\n","    \"\"\"Validation or test step.\"\"\"\n","    # Set model to eval mode\n","    model.eval()\n","    loss = 0.\n","    correct = 0\n","    y_preds = []\n","    y_targets = []\n","\n","    # Iterate over val batches\n","    with torch.no_grad():\n","        for i, (X, y) in enumerate(dataset.generate_batches()):\n","\n","            # Set device\n","            X, y = X.to(device), y.to(device)\n","\n","            # Forward pass\n","            _, logits = model(X)\n","            \n","            # Metrics\n","            loss += F.cross_entropy(logits, y, reduction='sum').item()\n","            y_pred = logits.max(dim=1)[1] \n","            correct += torch.eq(y_pred, y).sum().item()\n","\n","            # Outputs\n","            y_preds.extend(y_pred.cpu().numpy())\n","            y_targets.extend(y.cpu().numpy())\n","\n","    loss /= len(dataset)\n","    accuracy = 100. * correct / len(dataset)\n","    return y_preds, y_targets, loss, accuracy"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"NWK3QC-fyefM","colab_type":"code","colab":{}},"source":["def train(model, optimizer, scheduler, \n","          train_set, val_set, test_set, writer):\n","    # Epochs\n","    best_val_loss = np.inf\n","    for epoch in range(NUM_EPOCHS):\n","        # Steps\n","        train_loss, train_acc = train_step(model, DEVICE, train_set, optimizer)\n","        _, _, val_loss, val_acc = test_step(model, DEVICE, val_set)\n","\n","        # Metrics\n","        print (f\"Epoch: {epoch} | train_loss: {train_loss:.2f}, train_acc: {train_acc:.1f}, val_loss: {val_loss:.2f}, val_acc: {val_acc:.1f}\")\n","        writer.add_scalar(tag='training loss', scalar_value=train_loss, global_step=epoch)\n","        writer.add_scalar(tag='training accuracy', scalar_value=train_acc, global_step=epoch)\n","        writer.add_scalar(tag='validation loss', scalar_value=val_loss, global_step=epoch)\n","        writer.add_scalar(tag='validation accuracy', scalar_value=val_acc, global_step=epoch)\n","\n","        # Adjust learning rate\n","        scheduler.step(val_loss)\n","\n","        # Early stopping\n","        if val_loss < best_val_loss:\n","            best_val_loss = val_loss\n","            patience = PATIENCE # reset patience\n","            torch.save(model.state_dict(), MODEL_PATH)\n","        else:\n","            patience -= 1\n","        if not patience: # 0\n","            print (\"Stopping early!\")\n","            break"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"fhq8N-3qggmh","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"URGZGKyljabF","colab_type":"code","colab":{}},"source":["# Path to save model\n","MODEL_NAME = 'TextCNN'\n","MODEL_PATH = Path(f'models/{MODEL_NAME}.h5')\n","Path(MODEL_PATH.parent).mkdir(parents=True, exist_ok=True)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"hmoRnWEHid_B","colab_type":"code","colab":{}},"source":["# TensorBoard writer\n","log_dir = f'tensorboard/{MODEL_NAME}'\n","!rm -rf {log_dir} # remove if it already exists\n","writer = SummaryWriter(log_dir=log_dir)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"AbRtiX6_hbOi","colab_type":"code","outputId":"c44d04f5-dc39-4659-d371-26aea6725dcd","executionInfo":{"status":"ok","timestamp":1584480308228,"user_tz":420,"elapsed":44796,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":221}},"source":["# Training\n","train(model, optimizer, scheduler, \n","      train_set, val_set, test_set, writer)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Epoch: 0 | train_loss: 0.68, train_acc: 78.2, val_loss: 0.49, val_acc: 82.7\n","Epoch: 1 | train_loss: 0.44, train_acc: 84.6, val_loss: 0.44, val_acc: 84.6\n","Epoch: 2 | train_loss: 0.40, train_acc: 86.3, val_loss: 0.42, val_acc: 85.5\n","Epoch: 3 | train_loss: 0.36, train_acc: 87.4, val_loss: 0.40, val_acc: 86.1\n","Epoch: 4 | train_loss: 0.34, train_acc: 88.4, val_loss: 0.39, val_acc: 86.4\n","Epoch: 5 | train_loss: 0.31, train_acc: 89.2, val_loss: 0.39, val_acc: 86.6\n","Epoch: 6 | train_loss: 0.29, train_acc: 90.0, val_loss: 0.38, val_acc: 86.7\n","Epoch: 7 | train_loss: 0.27, train_acc: 90.8, val_loss: 0.38, val_acc: 86.8\n","Epoch: 8 | train_loss: 0.25, train_acc: 91.6, val_loss: 0.38, val_acc: 86.9\n","Epoch: 9 | train_loss: 0.23, train_acc: 92.3, val_loss: 0.38, val_acc: 86.9\n","Epoch: 10 | train_loss: 0.21, train_acc: 93.1, val_loss: 0.39, val_acc: 86.8\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"VOtoxvUPNDat","colab_type":"code","outputId":"d3e1b175-6f96-4232-8c20-3fec1cfc6ad5","executionInfo":{"status":"ok","timestamp":1584479545246,"user_tz":420,"elapsed":50661,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["%tensorboard --logdir {log_dir}"],"execution_count":0,"outputs":[{"output_type":"display_data","data":{"text/plain":["Reusing TensorBoard on port 6006 (pid 228), started 0:13:36 ago. (Use '!kill 228' to kill it.)"]},"metadata":{"tags":[]}},{"output_type":"display_data","data":{"text/html":["\n","    <div id=\"root\"></div>\n","    <script>\n","      (function() {\n","        window.TENSORBOARD_ENV = window.TENSORBOARD_ENV || {};\n","        window.TENSORBOARD_ENV[\"IN_COLAB\"] = true;\n","        document.querySelector(\"base\").href = \"https://localhost:6006\";\n","        function fixUpTensorboard(root) {\n","          const tftb = root.querySelector(\"tf-tensorboard\");\n","          // Disable the fragment manipulation behavior in Colab. Not\n","          // only is the behavior not useful (as the iframe's location\n","          // is not visible to the user), it causes TensorBoard's usage\n","          // of `window.replace` to navigate away from the page and to\n","          // the `localhost:<port>` URL specified by the base URI, which\n","          // in turn causes the frame to (likely) crash.\n","          tftb.removeAttribute(\"use-hash\");\n","        }\n","        function executeAllScripts(root) {\n","          // When `script` elements are inserted into the DOM by\n","          // assigning to an element's `innerHTML`, the scripts are not\n","          // executed. Thus, we manually re-insert these scripts so that\n","          // TensorBoard can initialize itself.\n","          for (const script of root.querySelectorAll(\"script\")) {\n","            const newScript = document.createElement(\"script\");\n","            newScript.type = script.type;\n","            newScript.textContent = script.textContent;\n","            root.appendChild(newScript);\n","            script.remove();\n","          }\n","        }\n","        function setHeight(root, height) {\n","          // We set the height dynamically after the TensorBoard UI has\n","          // been initialized. This avoids an intermediate state in\n","          // which the container plus the UI become taller than the\n","          // final width and cause the Colab output frame to be\n","          // permanently resized, eventually leading to an empty\n","          // vertical gap below the TensorBoard UI. It's not clear\n","          // exactly what causes this problematic intermediate state,\n","          // but setting the height late seems to fix it.\n","          root.style.height = `${height}px`;\n","        }\n","        const root = document.getElementById(\"root\");\n","        fetch(\".\")\n","          .then((x) => x.text())\n","          .then((html) => void (root.innerHTML = html))\n","          .then(() => fixUpTensorboard(root))\n","          .then(() => executeAllScripts(root))\n","          .then(() => setHeight(root, 800));\n","      })();\n","    </script>\n","  "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"4uarGMs2nSe5","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"6GXVrmFshbWn","colab_type":"code","colab":{}},"source":["import io\n","import itertools\n","import matplotlib.pyplot as plt\n","import seaborn as sns\n","from sklearn.metrics import classification_report\n","from sklearn.metrics import confusion_matrix\n","from sklearn.metrics import precision_recall_fscore_support"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"wNLdjRlChbVG","colab_type":"code","colab":{}},"source":["def plot_confusion_matrix(y_pred, y_target, classes, cmap=plt.cm.Blues):\n","    \"\"\"Plot a confusion matrix using ground truth and predictions.\"\"\"\n","    # Confusion matrix\n","    cm = confusion_matrix(y_target, y_pred)\n","    cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n","\n","    #  Figure\n","    fig = plt.figure()\n","    ax = fig.add_subplot(111)\n","    cax = ax.matshow(cm, cmap=plt.cm.Blues)\n","    fig.colorbar(cax)\n","\n","    # Axis\n","    plt.title(\"Confusion matrix\")\n","    plt.ylabel(\"True label\")\n","    plt.xlabel(\"Predicted label\")\n","    ax.set_xticklabels([''] + classes)\n","    ax.set_yticklabels([''] + classes)\n","    ax.xaxis.set_label_position('bottom') \n","    ax.xaxis.tick_bottom()\n","\n","    # Values\n","    thresh = cm.max() / 2.\n","    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n","        plt.text(j, i, f\"{cm[i, j]:d} ({cm_norm[i, j]*100:.1f}%)\",\n","                 horizontalalignment=\"center\",\n","                 color=\"white\" if cm[i, j] > thresh else \"black\")\n","\n","    # Display\n","    plt.show()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ROSikcLAnY37","colab_type":"code","colab":{}},"source":["def get_performance(y_pred, y_target, classes):\n","    \"\"\"Per-class performance metrics. \"\"\"\n","    performance = {'overall': {}, 'class': {}}\n","    metrics = precision_recall_fscore_support(y_target, y_pred)\n","\n","    # Overall performance\n","    performance['overall']['precision'] = np.mean(metrics[0])\n","    performance['overall']['recall'] = np.mean(metrics[1])\n","    performance['overall']['f1'] = np.mean(metrics[2])\n","    performance['overall']['num_samples'] = np.float64(np.sum(metrics[3]))\n","\n","    # Per-class performance\n","    for i in range(len(classes)):\n","        performance['class'][classes[i]] = {\n","            \"precision\": metrics[0][i],\n","            \"recall\": metrics[1][i],\n","            \"f1\": metrics[2][i],\n","            \"num_samples\": np.float64(metrics[3][i])\n","        }\n","\n","    return performance"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"StgvQqDWJ8Ta","colab_type":"code","outputId":"fd1c3ba5-1ea7-4063-f8cf-cd57b0bea3e4","executionInfo":{"status":"ok","timestamp":1584478731028,"user_tz":420,"elapsed":1325,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Test\n","y_preds, y_targets, test_loss, test_acc = test_step(model, DEVICE, test_set)\n","print (f\"test_loss: {test_loss:.2f}, test_acc: {test_acc:.1f}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["test_loss: 0.56, test_acc: 85.8\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"uYwfn_lgnf61","colab_type":"code","outputId":"93b4c99b-2401-42d2-b5e2-1d645200f321","executionInfo":{"status":"ok","timestamp":1584478731032,"user_tz":420,"elapsed":1316,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":595}},"source":["# Class performance\n","performance = get_performance(y_preds, y_targets, classes)\n","print (json.dumps(performance, indent=4))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["{\n","    \"overall\": {\n","        \"precision\": 0.8588907674416577,\n","        \"recall\": 0.8583333333333333,\n","        \"f1\": 0.8584737440288595,\n","        \"num_samples\": 18000.0\n","    },\n","    \"class\": {\n","        \"Business\": {\n","            \"precision\": 0.8334845735027223,\n","            \"recall\": 0.8164444444444444,\n","            \"f1\": 0.8248765154916928,\n","            \"num_samples\": 4500.0\n","        },\n","        \"Sci/Tech\": {\n","            \"precision\": 0.8220540540540541,\n","            \"recall\": 0.8448888888888889,\n","            \"f1\": 0.8333150684931507,\n","            \"num_samples\": 4500.0\n","        },\n","        \"Sports\": {\n","            \"precision\": 0.9189374856881154,\n","            \"recall\": 0.8917777777777778,\n","            \"f1\": 0.9051539415811436,\n","            \"num_samples\": 4500.0\n","        },\n","        \"World\": {\n","            \"precision\": 0.8610869565217392,\n","            \"recall\": 0.8802222222222222,\n","            \"f1\": 0.8705494505494505,\n","            \"num_samples\": 4500.0\n","        }\n","    }\n","}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"ESAY4v-onhqF","colab_type":"code","outputId":"88c8cb3b-e319-42c1-b806-49a0cfb39b8f","executionInfo":{"status":"ok","timestamp":1584478731032,"user_tz":420,"elapsed":1308,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":601}},"source":["# Confusion matrix\n","plt.rcParams[\"figure.figsize\"] = (7,7)\n","plot_confusion_matrix(y_preds, y_targets, classes)\n","print (classification_report(y_targets, y_preds))"],"execution_count":0,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAc8AAAGNCAYAAAB36PpMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd3wURRvA8d+TAoTepYlU6b2EXqV3\nXqQX6c0uIigqIAgCKk16kSqgSBUFRYr0Lr0XKaF3UiDJvH/s5khIh0sg5Pn62Y+3szOzs8vlnp3Z\nuT0xxqCUUkqpqHN53g1QSiml4hoNnkoppVQ0afBUSimlokmDp1JKKRVNGjyVUkqpaNLgqZRSSkWT\nBk+llFJxloi4isheEVlpr2cXke0iclJEFopIAjs9ob1+0t6eLVgd/e30YyJSKyr71eCplFIqLnsP\nOBJs/Rvge2NMLuAW0NlO7wzcstO/t/MhIvmBlkABoDYwQURcI9up6EMSlFJKOYtr8teM8fdxSl3G\n59pqY0zt8LaLSBZgFjAU+BBoAFwDMhhj/EWkLDDQGFNLRFbbr7eKiBtwGUgH9AMwxgyz63Tki6ht\nbs9+eEoppZTF+PuQME9zp9Tlu++HtJFkGQ30BZLZ62mA28YYf3v9ApDZfp0ZOA9gB9Y7dv7MwLZg\ndQYvEy4dtlVKKeVEAuLinAXSisiuYEs3x15E6gNXjTG7n8dRavBUKhIi4iEiK0Tkjoj8/Az1tBGR\nNc5s2/MiIhVF5Njzbod66V03xpQMtkwJtq080FBEzgILgGrAGCClPSwLkAW4aL++CLwKYG9PAdwI\nnh5GmXBp8FQvDRFpbV+d3hcRLxH5XUQqOKHqZsArQBpjzJtPW4kxZp4xpqYT2hOjRMSISK6I8hhj\n/jHG5ImtNqk4RAAR5ywRMMb0N8ZkMcZkw5rw87cxpg2wDutvFqADsMx+vdxex97+t7Em/SwHWtqz\ncbMDuYEdkR2m3vNULwUR+RDrxn8PYDXwEGvmXCNg0zNW/xpwPNh9lHhNRNz0XKgIyXPtl30CLBCR\nIcBeYLqdPh2YIyIngZtYARdjzCERWQQcBvyB3saYgMh2oj1PFeeJSApgMNab/ldjzANjzCNjzApj\nzMd2noQiMlpELtnLaBFJaG+rIiIXROQjEblq91o72tsGAV8ALewebWcRGSgic4PtP5vdW3Oz198S\nkdMick9EzohIm2Dpm4KVKyciO+3h4J0iUi7YtvUi8pWIbLbrWSMiYU6eCNb+vsHa31hE6orIcRG5\nKSKfBstfWkS2ishtO+/4YN+F22hn+9c+3hbB6v9ERC4DM4PS7DI57X0Ut9czicg1EanyTP+wSkWR\nMWa9Maa+/fq0Maa0MSaXMeZNY4yfne5rr+eyt58OVn6oMSanMSaPMeb3qOxTg6d6GZQFEgFLIsjz\nGVAGKAoUAUoDA4Jtz4B1DyQz1vfBfhCRVMaYL4GvgYXGmKTGmOlEQESSAGOBOsaYZEA5YF8Y+VID\nv9l50wDfAb+JSJpg2VoDHYH0QAKgTwS7zoB1DjJjBfupQFugBFAR+NwekgIIAD4A0mKdu+pALwBj\nTCU7TxH7eBcGqz81Vi/cMWnDLnMK62p/rogkBmYCs4wx6yNor3qZxcKw7fOmwVO9DNJgTSyIaCix\nDTDYGHPVGHMNGAS0C7b9kb39kTFmFXAfeNp7eoFAQRHxMMZ4GWMOhZGnHnDCGDPHGONvjPkJOIr1\nPbUgM40xx40xPsAirMAfnkfAUGPMI6zJE2mBMcaYe/b+D2NdNGCM2W2M2Wbv9ywwGagchWP60hjj\nZ7cnBGPMVOAksB3IiHWxouIlp862fWG92K1TKmpuYE1pj+gefibgXLD1c3aao44ngq83kDS6DTHG\nPABaYN179RKR30QkbxTaE9Sm4N8vuxyN9twIdp8mKLhdCbbdJ6i8iLwuIitF5LKI3MXqWUf2fbpr\nxhjfSPJMBQoC44KGypR6WWnwVC+DrYAf0DiCPJewhhyDZLXTnsYDIHGw9QzBNxpjVhtjamD1wI5i\nBZXI2hPUpkinyDvBRKx25TbGJAc+xZojGZEIH0UmIkmxvrA+HRhoD0ur+EqHbZV68Rlj7mDd5/vB\nniiTWETcRaSOiIyws/0EDBCRdPbEmy+AueHVGYl9QCURyWpPVuoftEFEXhGRRva9Tz+s4d/AMOpY\nBbxuf73GTURaAPmBlU/ZpuhIBtwF7tu94p5PbL8C5IhmnWOAXcaYLlj3cic9cytV3CTosK1ScYUx\n5lusZ1sOwHq25XngbWCpnWUIsAvYDxwA9thpT7OvP4GFdl27CRnwXOx2XMKaDl+Z0MEJY8wNoD7w\nEdawc1+gvjHm+tO0KZr6YE1GuofVK174xPaBwCx7Nm6kz1kTkUZYXwsKOs4PgeJBs4yVehnpg+GV\nUko5jUvSjCZhoQ6RZ4wC323f7DbGlHRKZU6mD0lQSinlXC/4kKszvPxHqJRSSjmZ9jyVUko51ws+\nU9YZNHgqpZRyItFhW6WUUkqFpj1PpZRSzhP0k2QvOQ2eSimlnEuHbZVSSin1JO15KqWUcqL4MWFI\ng6dSSinncnn573m+/JcHSimllJNpz1MppZTzBP2qyktOg6dSSinnigdfVXn5Lw+UUkopJ9Oep1JK\nKSfS2bZKKaVU9OmwrVJKKaWepD1PpZRSzqXDtkoppVQ0iMSLYVsNnkoppZxLe54KQBIkMeKR+nk3\nI84pmD39825CnOTu+vJftceUwOfdgDjov3NnuXH9ur7pokmDZxSIR2oSlnn/eTcjzlk5v/fzbkKc\nlD5FoufdhDjrob+Gz+iqXL608yvVYVullFIqOuLH9zxf/iNUSimlnEx7nkoppZxLh22VUkqpaIgn\nv6ry8h+hUkop5WTa81RKKeVE8WPCkAZPpZRSzhUP7nm+/JcHSimllJNpz1MppZRz6bCtUkopFU06\nbKuUUkqpJ2nwVEop5Txiz7Z1xhLhbiSRiOwQkX9F5JCIDLLTfxSRMyKyz16K2ukiImNF5KSI7BeR\n4sHq6iAiJ+ylQ1QOU4dtlVJKOVfsDNv6AdWMMfdFxB3YJCK/29s+Nsb88kT+OkBue/EEJgKeIpIa\n+BIoCRhgt4gsN8bcimjn2vNUSikV5xjLfXvV3V5MBEUaAbPtctuAlCKSEagF/GmMuWkHzD+B2pHt\nX4OnUkoppxIRpyxAWhHZFWzp9sR+XEVkH3AVKwButzcNtYdmvxeRhHZaZuB8sOIX7LTw0iOkw7ZK\nKaWcRiAo8DnDdWNMyfA2GmMCgKIikhJYIiIFgf7AZSABMAX4BBjsrAYF0Z6nUkqpOM0YcxtYB9Q2\nxnjZQ7N+wEwg6Ne+LwKvBiuWxU4LLz1CGjyVUko5jzhxiWg3IunsHici4gHUAI7a9zERq/vbGDho\nF1kOtLdn3ZYB7hhjvIDVQE0RSSUiqYCadlqEdNhWKaWUE4kzh20jkhGYJSKuWB3BRcaYlSLyt4ik\nwwq/+4Aedv5VQF3gJOANdAQwxtwUka+AnXa+wcaYm5HtXIOnUkqpOMcYsx8oFkZ6tXDyG6B3ONtm\nADOis38NnkoppZwqlnqez5UGT6WUUk4VH4KnThhSSimlokl7nkoppZwqPvQ8NXgqpZRynih8zeRl\noMO2SimlVDRpz1MppZTTSOx9z/O50uCplFLKqeJD8NRhW6WUUiqatOeplFLKqeJDz1ODp1JKKaeK\nD8FTh22fUkJ3V/4Z25rtE9uxe0oHBrQrF2L7wLfKs396R/ZOfYtejazHL37QrCTbJrRj24R27Jrc\ngfurPiBVskSOMi4uwtYf2rF4cONw9zuyRxXKF7R+p7VK0axsGd+WbRPasfbbluTIlBKA8gUzs2V8\nW+6t+oAmFXKHW5e7mwvj36vB/ukd2TetI43tvD0bFmPX5A4s+aoJ7m7WW6RcgcyM6F7FUTZtCg+W\nDW0ajTMWtvLF8lCzYknqVPGkfvXyjvShX/anWpki1KpUim7tm3Pnzm3HtiOHDtC4dmXeKF+cmhVL\n4uvrG2bdPTq24r+zZwAYMfRLyhTORb7X0obIs33LJupWLUuOV5Ly2/Jfw6zn/r171Kni6ViKvp6F\nQZ/1AWDm1AnUqFCCDi0b8/DhQwB2btvM4M8+dpS/cf0a7Zs3fIqzE33jx46hRNGCFC9SgHFjRofY\nNmH8OIoUzEvxIgX4tF/fMMt7eXnRtFF9ANb+9SflSpegZNFClCtdgvXr/g6zzOJffqZ4kQIkTuDC\n7l27HOlbNm+mVLHClPcsyckTJwC4ffs29evUJDAw0JGvbq03uHXr1jMdd3RdOH+e+rWqU7pYQTyL\nF2Li+LGObTdv3qRRvZoUK5iHRvVqhmrb7l07SZ00AUt//SXMun18fKhboyoBAQGOtLt375IvZ1b6\nvP9OmGWGDRlE3hyvUsGzOBU8i7Pmj1UAbNuymXKlilK5fGlOnXx8DhvXrxXiHDasG7qdKmZp8HxK\nfo8CqN33Zzx7zsGz5xxqlsxG6bwZAWhXswBZ0iWjSJeZFOv6Iz+vPwrA97/sokyvOZTpNYcvZvzD\nPwcucOve4w/+txsX59j5G+HuM3WyRJTOm5HNB62fmhv7TnU6frOKMr3msHDdEfq18gTg/LV7dPv2\nDxauOxLhMXzSqgzXbntTuPNMinWdyT/7LwDQslpeSvWYxbbDl6hRIhsA/VqXYdj8bY6y1+/4cPnm\nA8rmzxS9ExeGBUv/4Pf121m5drMjrWKV6qzZtJvVG3eSPWduJoweCYC/vz/v9+zE16PG8dfmPSxc\nthp3d/dQdR4/epjAgACyZssOwBu16rJszT+h8mXK8irfjp9Co/+1CLd9SZMl4/f12x1L5ixZqV3P\nusBZ9ssCVm/cSYlSZdj4958YYxj77XDe7dPfUT5N2nSkfyUDO7dveboTFEWHDh5k5oyp/LNlBzt2\n/8vvq1Zy6uRJADasX8fKFcvYsftf9vx7iPc/7BNmHWNHf0fHzl2tdqdJyy9LV7Br3wGmzphFp7fa\nhVmmQIGCLFj0KxUqVgqRPmb0tyxZsYoR345m6pRJAAz/egh9+32Ki8vjj57WbdoxZdKEZz7+6HBz\nc2PI8JHs2HuQvzZsYerkCRw9chiA70d9Q+Uq1dl78BiVq1Tn+1HfOMoFBATw5YD+VHujRrh1z501\nkwaNmuDq6upIGzroC8pVqBhhm3q98z6btu9h0/Y91KxdF4BxY77j5yUrGT7iO2ZMnQzAyOFD+ahv\n/xDnsGXrNkybMjH6JyImxNJPkj1vGjyfwQPfR4DVg3NzdcF6aD90q1+Er+dtw17l2h2fUGWbV83L\nIjuoAmROm5TapbMz8/cD4e6vcYXcrNl11rFuDCRPnACA5EkS4nXzAQD/XbnLwTPXCQw0Eba/Q62C\njFyw3VHXjbtWO0UEdzdXEidy51FAIK2q52PNrjMhAj3Aii0naVEtX4T7eFqVqr6Bm5t1V6FYydJ4\nXbIuGDau+4u8+QuSv2BhAFKlThPiQyrI0l8WUKNOA8d68ZKevJIhY6h8r2Z9jXwFCoX4IIrI6ZMn\nuHH9KqXLWr1kYwyPHj3C18cbN3d3lvz8E1Wq1yRlqtQhytWs04ClvyyM0j6e1tGjRyhVypPEiRPj\n5uZGxUqVWbrU6k1PmTyRPn37kTBhQgDSp08fZh1LlyymZq3aABQtVoxMmayLo/wFCuDr44Ofn1+o\nMnnz5eP1PHlCpbu7u+Pj7Y2Pjzfu7u6cPnWKCxfOU6lylRD56jVoyKKFPz31cT+NDBkzUrRYcQCS\nJUtGnrx5uWS/x1atXE7rtu0BaN22Pb+tWOYoN3nCeBo1bkq6dGGfP4BFC+ZTt8HjkYa9e3Zz9eqV\nCANueNzd3fHx8cbbfn+dPn2KixfOU7FSlRD56tRryOJFC6Jdf0wREacsL7IYC54iEiAi+0TkXxHZ\nIyLlIi8VZj09RKS9s9vnDC4uwrYJ7fhvYU/+3nuOnccuA5A9Y0qaVc7DpnFtWDqkKTnt4dQgHgnd\nqFEyG0s3nXCkjexRlc+mbSTQhB/wyhbIzN4TVxzrvUavYcmQppyc243W1fMzauGOKLc9RRLrQ/TL\nDuXZMr4t8z6rT/qUiQGYuHwvG0a35tV0ydh66CLtaxZk0vJ9oerYc/yKYwj5qYnQtlkD6lUrx/xZ\n08PMsmjebKpUrwXAmVMnEBHavdmAulXLMmnst2GW2bV9K4WKhPq1ome2YsnP1G/czPGH3aFLTxrX\nrszFC+cpWbosi+bPpn3nHqHKFS5WnJ3bNodKd6YCBQqyefM/3LhxA29vb/74fRUXzp8H4OTx42ze\n9A8Vy3lSo1pldu3cGar82TNnSJUylSPABrfk18UULVY8zG3h+bhvfzp3bM/Ib4bRo9fbfPnFZwwc\nNCRUvlSpUuHn58eNG+GPusSkc+fOsn/fPkqWskZurl29QoaM1oXWKxkycO2q9Td36eJFVi5fSudu\nof99gzx8+JCzZ0/z2mvZAAgMDGRAv48ZMmxkpO2YOukHypUqSu/unR1DsB9+3I/und/i+5Hf0K1H\nb776cgCfD/wqVNmgc3jzOZ3D+Cgme54+xpiixpgiQH9g2NNUYoyZZIyZ7dymOUdgoKFMrznkajOF\nknkykP+1NIB1P9TvoT8V3pnHzN/3M/mjWiHK1SuTk62HLjl6cnU8c3D1tjd7T16NcH8ZUifherBe\n7DtNStBkwK/kajuFOWsO8k23KlFuu5urC1nSJWPb4UuUe3su2494MaxrZQB+WnuEsr3n0GnE77zT\ntAQTlu6lVqnszB/QgBHdqxB0QXj1tjcZ0ySN8j7Dsvi3taxat5VZC5cye8Zktm/ZFGL7uO++wc3N\nlSZvtgSsYdud27cwZtJMFv+2lj9WLWfTxnWh6r165TKp06QNlf6sli/5mUZNmzvWmzZvze/rtjFm\n0kymTxpHx269WP/Xanp0bMXgzz523JdKkzY9Vy57Ob09weXNl4+P+nxCgzo1aVivNkWKFHX0yv0D\n/Ll58yYbN2/j6+Ejadu6uWOkJIiXlxdp06ULVe/hQ4cY8OknjJ8wOVrtKVK0KBs3b2P1X+s4e+Y0\nGTJkxBhD29Yt6Ni+LVeuPL4QTJcuPV6XLj3FUT+b+/fv067Vmwwb+R3JkycPtV1ECHrD9/v4AwYN\nGRbhKMWN69dJkeLxxfK0yROpUasOmbNkibAdnbv2YN/hE2zavodXMmRkQD9rWL1wkaKs3biFlavX\ncvbs43P4VtuWdO3YjqtPnkOv2D+HTwp6SIL2PJ0jOXALQESqiMjKoA0iMl5E3rJfDxeRwyKyX0RG\n2WkDRaSP/Xq9iHwjIjtE5LiIVLTTXUVkpIjstMt2t9MzishGuwd8UEQq2nl/tNcPiMgHz3pwdx74\nseHf89QsZd1fu3j9vqNXuWzzSQpmD/mB9GblPI77oABl82eifpmcHJ3Vhdn961OlSFZm9K0Taj8+\nD/1JmMD6MEybwoNCOdI5eru/bDhGmWjcf7xx14cHvo9Yutlq56//HKdo7pBDURlTJ6Fkngys2HqS\n9/5XgrZfr+T2fT+qFnsNgEQJXPH184/yPsOSIaPVc02bLj216jZk357HPaKff5rD2jWrGDPpR8cf\nUsZMmfEsW4HUadLikTgxVd+ozcF/94aqN5GHR5hDjM/i8MH9BPj7U6ho8VDbrnhdYt+eXdSq25Cp\nE8bww7S5JE+Rks12YPfz9SVRokShyjnbW506s2XHbv5at5GUqVKRO/frAGTOnIXGTZoiIpQqXRoX\nFxeuX78eoqyHh0eoyVcXLlygxZtNmDZjNjly5nyqNhljGP71EPp/9jlDvxrE0GEj6NSlKxOCTdLx\n8/XFw8Pjqep/Wo8ePaJdq2Y0b9Gaho0fT35Ll/4VLntZFzqXvbwcQ7R79+ymU/vWFMqTg2VLFvPR\n+2+zcvnSEHUm8vDAL9g53LF9K1Mn/UChPDkY0L8vC+bP4csB/XlS+ldewdXVFRcXFzp06sLuXSFH\nBowxjBw+lI/7D2D40MEMHvoNHTp1YdKEcY48vn6+JIrlcxgeDZ7PxsMOWkeBaUDosYZgRCQN0AQo\nYIwpDIQe37G4GWNKA+8DX9ppnYE7xphSQCmgq4hkB1oDq40xRYEiwD6gKJDZGFPQGFMImBlOe7qJ\nyC4R2WUePgi1PW0KD8fQZ6IEblQv/hrHzt8ErHuBlYtkBaBi4SycvPB4FlzyxAmoUDgLK7acdKR9\nMXMTudpOIW+HabQftpL1//5HpxG/h9rnsf9ukjNTKgBu3fMleZKE5MpsrVcr/lqEk43CsmrbKSoV\nfhWwZu4ePRey/BcdyvPVbGuSi0dCN4wxBBpD4oTWvcjcmVNx6FzID+Do8H7wgPv37jleb1z/F3ny\nFQBg/do1TBr3HdPn/oJH4sSOMpWr1eDo4UP4eHvj7+/P9i3/kDtP6PuuuXLn4dyZU0/dtrAs/3UR\nDYP1OoMbNXwwH/b7HABfXx/rj9/FBR9vb8Aabg46tph09ao1evHff/+xbOmvtGjVGoAGDRuzYb0V\nyE8cP87Dhw9JmzZkzzz3669z7txZx/rt27dp2rAeXw0dTrny5Xla8+bMplbtuqROnRpvH29cXFwQ\nFxe87XNjjOHylcu8li3bU+8juowxvN2jC3ny5OPt90JeP9ep14D5c63BrvlzZ1O3vnX/8sDRUxw4\ndpoDx07TqMn/+Hb0eOo3DDkzPlWqVAQEBDguQqb9OJdDJ85y4NhphgwbQcvW7Rg0JPQgXFCwBli5\nbCn58od8r/w0bzY1a1nn0MfbOocu8vj9ZYzh6uXLjuFiFfNiY9g2L1AbmC0RX0rcAXyB6SLSFPAO\nJ1/Q9wl2A9ns1zWB9iKyD9gOpAFyAzuBjiIyEChkjLkHnAZyiMg4EakN3A1rJ8aYKcaYksaYkpIg\nSajtGVIn4Y8RzdkxsT2bxrVh7Z5z/L79NACjFu6gcYXc7JzUnq86VqTn6DWOcg3L52bt7nN4P0WP\n7Y8dp6lU2Br+CQg09B69hp8+b8D2ie1oXT0//aduBKDE669wcm43mlbKw7j3arB7SgdHHdsmPJ4x\nOWD6Rga0K8eOie1pXT0f/aZscGwrktO62t5nDyUvXHeUXZM7ULZAJsekpcpFs/LH9jPRPo4g169d\npVn96tSuXJqGNStSrUYdqlSvCcAX/T7gwf17tG1WnzpVPPn0I2uKf4qUqejS810a1KhAnSqeFCxc\nlOo1Q/fSq9Wsw9bNGx3rXw/8FM9COfHx9sazUE6+/8a6Nvt3zy48C+Xkt+W/8ulH7/BG+ce9yjpV\nPEPUuXLZYhr+L3TwPLjfuh8cdI+10f9aULNiSXbv2Epl+3i2bNpA1Rq1n/pcRVWr5v+jWOH8NGvc\ngNFjfyBlSmsIsUPHTpw5fZoSRQvSvk1Lps2YFerKPkmSJOTIkdMxQ3fShPGcOnWSYUMG41miKJ4l\nijqCc89uXRxfS1m2dAk5s2Vh+7atNG1UjwZ1H9+m8Pb2Zs7sH+nRqzcA777/IU0a1KXvh+/T1b53\nuGf3bkp7lnFMEIsN27ZsZsH8uWzcsC7U10M+7PMJ6/7+i2IF87B+3Vo+6PNJtOqu+kYNtj5x+yEs\nb/fsyp7d1jn84rNPKFuyCOVKFeWfjesYNuI7Rz5vb2/mz5lN1x69AOj97ge82aQ+/fp+SKeu3QGr\nV1yytGesnsMIxYPZtvLkfQ+nVSxy3xiTNNj6FaAQ8DrwqTGmrp0+DdhkjPlRRBIC1YFmQDZjTDU7\n8N03xowSkfVAH2PMLhFJC+wyxmQTkcXAFGPM6jDakQmoB/QGvjPGzBaRpEAtoB1w0xjTKaJjcUnx\nqklY5v1nPCPOsfbbljT9Ygl3Hjh3SPJp/DmqBW8OXMrt+2G35dj83rHcosd8fXxo2bgWi1etC3M2\nbmx7s/4bTJv7MylSpoo0b/oUMT+8G55lS5ewd89uBg4Ob+DH+T764D3qN2hI1WrVn7muh/6BkWeK\nYfv27mHCuNFMmRF7UzU++eh96tRvQJWq0T+HlcuXZu/uXU4LVe7pcprUjYY7pa6r05vvNsaUdEpl\nThYr9zxFJC/gCtwAzgH5RSShiKTECpbYAS2FMWYV8AHWMGtUrQZ6ioi7XdfrIpJERF4DrhhjpmIN\nHRe3g66LMWYxMAAIfQPrBdZvynpeTZ/seTeDtCk8GLt4d7iB83lL5OHBB598zmWvi8+7Kdy4fo0u\nvd6NUuB83ho1bhLrQ38FChR0SuB8URQtVpyKlauEeEhCTMtXoMBTBU719GKyj+9hD6OC1QHvYIwJ\nAM6LyCLgIHAGCJrtkQxYJiKJ7PwfRmNf07CGcPfYQ8PXgMZAFeBjEXkE3AfaA5mBmSISdOEQ+u79\nCyxogtDzdv2ODyu2now843NUuVr0v1cXE9KkTUeturHzhCFn6Ni5S6zur1OXrrG6v9jQrkOEg1lO\n91anF+scvuiTfZwhxoKnMSbcsTJjTF8grOeDlQ4j78Bgr6sEe30d+56nMSYQ+NRegptlL0+KU71N\npZSKS+JD8NQnDCmllFLR9IJMzVJKKfUyCHpIwstOg6dSSinnevljpw7bKqWUUtGlPU+llFLOI/Fj\nwpAGT6WUUk4VH4KnDtsqpZRS0aQ9T6WUUk4VH3qeGjyVUko518sfOzV4KqWUcq740PPUe55KKaVU\nNGnPUymllNOI6BOGlFJKqWiLD8FTh22VUkqpaNKep1JKKaeKDz1PDZ5KKaWc6+WPnTpsq5RSKu4R\nkUQiskNE/hWRQyIyyE7PLiLbReSkiCwUkQR2ekJ7/aS9PVuwuvrb6cdEpFZU9q/BUymllFMFzbh9\n1iUSfkA1Y0wRoChQW0TKAN8A3xtjcgG3gM52/s7ALTv9ezsfIpIfaAkUAGoDE0TENbKda/BUSinl\nPBI7wdNY7tur7vZigGrAL3b6LKCx/bqRvY69vbpYO2kELDDG+BljzgAngdKRHaYGT6WUUnGSiLiK\nyD7gKvAncAq4bYzxt7NcADLbrzMD5wHs7XeANMHTwygTLp0wpJRSymkEcOJk27QisivY+hRjzJSg\nFWNMAFBURFICS4C8TttzJDR4KqWUciKnPmHoujGmZGSZjDG3RWQdUBZIKSJudu8yC3DRznYReBW4\nICJuQArgRrD0IMHLhEuHbbKLTgEAACAASURBVJVSSsU5IpLO7nEiIh5ADeAIsA5oZmfrACyzXy+3\n17G3/22MMXZ6S3s2bnYgN7Ajsv1rz1MppZRTxdIzEjICs+yZsS7AImPMShE5DCwQkSHAXmC6nX86\nMEdETgI3sWbYYow5JCKLgMOAP9DbHg6OkAZPpZRSThUbTxgyxuwHioWRfpowZssaY3yBN8Opaygw\nNDr712FbpZRSKpq056mUUsp5JNaGbZ8rDZ5KKaWcRgAXl5c/euqwrVJKKRVN2vNUSinlVDpsq5RS\nSkVTfPg9Tx22VUoppaJJe55KKaWcR2fbqiCFcqRnzc/vPu9mxDnZ6g953k2Ik279Peh5N0Gpp2Y9\nGP7lj546bKuUUkpFk/Y8lVJKOZFTf1XlhaXBUymllFPFg9ipw7ZKKaVUdGnPUymllFPpsK1SSikV\nHfHkqyo6bKuUUkpFk/Y8lVJKOU18+Z6nBk+llFJOFQ9ipw7bKqWUUtGlPU+llFJOpcO2SimlVDTF\ng9ipw7ZKKaVUdGnPUymllPOIDtsqpZRS0WJ9VeV5tyLm6bCtUkopFU3a81RKKeVE+pNkSimlVLTF\ng9ipw7ZKKaVUdGnPUymllFPpsK1SSikVHfqTZEoppZQKi/Y8lVJKOY3+JJlSSin1FOJD8NRhW6WU\nUiqatOeplFLKqeJBx1ODp1JKKefSYVullFJKhaI9T6WUUs4TT77nqcFTKaWU04g+GF4ppZSKvngQ\nO/Wep1JKqbhHRF4VkXUiclhEDonIe3b6QBG5KCL77KVusDL9ReSkiBwTkVrB0mvbaSdFpF9U9q/B\n08l8fX2pXbUc1cqXoJJnEUZ8Pcix7Z/1f1OjYmmqVyhJw1pVOHPqJAAXzv9H0/o1eKNCKaqWK85f\na34Ps+4rl71o27wxAI8ePeKdHp2oUrYYFUsVYuy334RZxhjDsMGfU654fiqWKsS0SeMBWLnsVyp5\nFqFR7arcvHkDgLOnT9HtrdaOsg8fPqRxnWr4+/s/+4mxJUzgxj+Tu7J9Rk92z+rNgI5VHduqFM/O\nlmnd2Ta9B2vHdyJH5tQAJHB3Zc7ANzk4/102TupK1gwpAahWMgebp3Zn54+92Dy1O5WLZw93v/MH\nNydbxlQANK9ekJ0/9mLHzJ4sG9mWNCkSh8j7Xoty+GwcFCo9yP11X7Jteg+2Te/Bz8NaOdJnfv4/\ndszsyaCu1R1pn7SvRIMKeR3rdcq+zuedquJM3bt0Imum9JQoWjBE+s2bN6lXuwYF8+WmXu0a3Lp1\nC4Cf5s+jVLHClCxaiCoVy7H/33/DrNcYQ+0a1bh79y7nz5+n1htVKVY4P8WLFGD82DFhllmxfBml\nihXGs0RRynuWZPOmTQAcP3aMcqVLUKpYYbZt3QqAv78/dWu9gbe3t6N8uzYtOXnixDOfk8hcOH+e\n+rWqU7pYQTyLF2Li+LGObTdv3qRRvZoUK5iHRvVqOs7bnTt3aPG/hpQvXQzP4oWYO3tmmHX7+PhQ\nt0ZVAgICHGl3794lX86s9Hn/nXDbNHnCeEoWyY9n8UJ8/uknAGzbsplypYpSuXxpTp20zsvt27dp\nXL8WgYGBjrIN6z5u54vARcQpSyT8gY+MMfmBMkBvEclvb/veGFPUXlYB2NtaAgWA2sAEEXEVEVfg\nB6AOkB9oFaye8I/xaU6MCl/ChAlZvGINf2/ezdpNu1j31xp279wOwCcfvs0P02axdtMumjRryfej\nhgEweuQwGjZuxl+bdjJpxlz6ffRumHVPGj+Gth06AbBi6S889PNj/da9rN6wndk/TuO/c2dDlVkw\nbzYXL15g066D/LPzAI3+1xyA6VMm8Me6rbTr2IVff14AwPAhX9Lv88fBPkGCBFSoXJVlvy5y2vnx\ne+hP7fdn4dlpIp6dJlLTMxel82cBYOxH9en41WLKdJ7Ewr8O0K99JQDeqlecW/d8KNh6LOMWbWVo\njxoA3LjjTbN+8yn11gS6fr2EGZ81DXOf+bKlw9XFhbNet3B1dWHku3Wo/d6PlO44kYOnrtCjaWlH\n3izpk1O9VE7+u3w73GPw8XtEmc6TKNN5Em/2/wmAgjlewcfvEaU7TqRE3swkT5KQDGmSUipfFlZs\nOuoo+/vW49QtlwePhO7PdiKDadfhLZat/CNU+qgRw6lSrToHj5ygSrXqjBoxHIBs2bKz5u8N7Np3\ngP6ffU7vnt3CrPeP31dRqHARkidPjpubG8NHfMve/YfZsGkbkyf9wJHDh0OVqVqtOjv2/Mv23fuY\nNHUGvXp0AWDa1MmM/G4MS1asYvT3owCYMmkirVq3JXHixxcp3br35LtRI575nETGzc2NIcNHsmPv\nQf7asIWpkydw9Ih1PN+P+obKVaqz9+AxKlepzvejrAvTqZMnkCdvfjbv2Mtvq//ms34f8/Dhw1B1\nz501kwaNmuDq6upIGzroC8pVqBhuezZuWMdvK5ezecdetu85wLvvfwTAuDHf8fOSlQwf8R0zpk4G\nYOTwoXzUtz8uLo8/vlu2bsO0KROf/cQ4iYhzlogYY7yMMXvs1/eAI0DmCIo0AhYYY/yMMWeAk0Bp\nezlpjDltjHkILLDzRkiDp5OJCEmSJgWs3qH/o0eOm+ciwv179wC4d/cOGTJkdKTfu3fXTr/rSH/S\nb8uXUPWNWo4y3t4P8Pf3x9fXhwTu7iRLljxUmVnTJ/NR388cf2jp0qUHwEVceOjnh4+3D+5u7mzb\nson0r2QgR87cIcrXqdeQxYsWPNM5edIDH+sDx93NFTc3F4wxABgDyRMnBCB5kkR4XbfOVf0KeZn3\nxz4Aft1wmCp2D/PfE5fxumHlOXzmKokSupHA3ZUntaxR2BHAgp67mSSRFbySJUno2A/AiLdr89nE\nNY42RdWjgAA8ErojIri7uRAQaPi8UzWGzFwXKu8/+85St9zr0ao/IhUqViJ16tSh0leuWEbbdh0A\naNuuAyuWLwWgbLlypEpl9cJLe5bh4sULYda74Kd5NGhofYZkzJiRYsWLA5AsWTLy5s3HpUsXQ5VJ\nmjSp4/3+4MEDx2t3d3d8fLzx8fbG3d2d27dvs+q3FbRp1z5E+fIVKvL33385dbQjLBkyZqRoscfH\nkydvXsfxrFq5nNZtrXa1btue31YsA+y/3/v3MMZw/8F9UqVKjZtb6GkjixbMp26Dho71vXt2c/Xq\nFaq9USPc9kyfMokP+vQlYULr/Z8uvfV3GnTevH28cXN35/TpU1y8cJ6KlaqEKB8Tf6cviLQisivY\nEuaVnohkA4oB2+2kt0Vkv4jMEJFUdlpm4HywYhfstPDSIxSrE4ZE5DOgNRAABALdjTHbw8hXEmhv\njHnXXncHjgN37CwZ7Dqu2eul7SuGyPZfBehjjKn/jIcSoYCAAGpW9uTM6VN07NKD4iWtns234ybT\npllDEnl4kDRZMlb9ZQ1p9en/OS2a1GXGlAl4P3jAomWhexHnzp4hZcqUjj+u+o3+xx+/raDw61nx\n8fFm8NejSBXGB+i5M6dZ9uvPrFq5jDRp0zF0xHfkyJmbdz/sS/NGtXklY0Z+mDKLrh1aMWnG3FDl\n8+YvyL49u5x5enBxEbZM7U7OzKmZvHQnO49YH1q9RixjyYi2+Po94q63H5V7TAMgU9pkXLhqXVwE\nBARy94EfaVIk5sadx8N9TSrnZ99xLx4+Cgi1v7KFsrJo7QEA/AMCee/blez8sRcPfB9x6sIN3v/+\nNwDqV8jDpev3OHDqSoTtT5TAjU1TuhEQEMioeZtYsekox85d5/rtB2yd1p2f1uwnZ+bUuLgI+457\nhSq/59hFyhd+jcXrDj3F2Yu6q1eukDGjdSGWIUMGrl4JfVw/zpxOrVp1wiy/dctmxk+YHCr93Nmz\n7Nu3l1KlPcMst2zpEr4Y0J9rV6/y6zLr3Hbv2ZsuHdvj5+fH+AmTGTb0K/r2+zRE7wnAxcWFnDlz\nsf/ffyleokS0jvdpnTt3lv379lGylHU8165eIYN93l7JkIFrV63z1q1Hb1o1a0yeHFm4f+8eM+f8\nFKr9Dx8+5OzZ07z2WjYAAgMDGdDvY6bMmM36dX+F24ZTJ0+wdfMmvvrycxIlSsRXw0ZQomQpPvy4\nH907v4WHhweTp89iQP+P+XzgV6HKp0qVCj8/P27euEHqNGmccFaentVrdNqMoevGmJIR70+SAouB\n940xd0VkIvAVYOz/fwt0claDgsRaz1NEygL1geLGmMLAG4SM9g7GmF1BgdNWAVgZNIYNTCLkmHak\ngTM2ubq6snbTLvYePsPePbs4cvggAFN+GMO8X5az98gZWrbpwJeffgzAkl8W0qJ1e/YeOcO8X5bz\ndve3QtzPALh65TJp0qZzrO/dvRNXV1f+PXaOHfuPM2n895w7czpUW/we+pEwUSLWbNhG2w6d+KC3\ndeFWudobrNm4nTkLl/LHquVUr1mb06dO0LldCz56p4fjPpSrqysJEiRw9JidITDQUKbzJHI1+46S\neTOTP7t1lf1O87I06TuXXM2+Y86qfXzzdq1IarLky5aOIT1q8PaoFWFuz5AmKddvW8fj5upC18al\nKNN5EjmajOLgqSt83LYiHgnd6du2EoOn/x3p/vI0/54K3abQYfBiRr5Tm+yZrAvbj8f9QZnOkxiz\ncAtfdK7G4Gl/07ddJeYOfJOO9R8Hgqu3HpAxbbIoHZuziIT++sCG9euYNXM6Q4aFfb/81s2bJEsW\nsp3379+nVfP/MfLb0SRPHnqkA6BR4yb8e/AoixYvZfDAzwHImjUra9auZ8OmrSROnJiLFy+QJ28+\nOnVoR9vWLThx/LijfLp06fHyuvQshxtl9+/fp12rNxk28rswj0eCjR+u/XM1hQoX4djpC/yzfQ99\nPniXu3fvhsh/4/p1UqRI6VifNnkiNWrVIXOWLBG2w9/fn1s3b7J24xa++vob3mrbEmMMhYsUZe3G\nLaxcvZazZ0+TIUNGjDG81bYlXTu2C3FBFJvnLTIu4pwlMnbHajEwzxjzK4Ax5ooxJsAYEwhMxRqW\nBbgIvBqseBY7Lbz0iI8x8uY5TUasqwg/AGPMdWPMJREpJSJbRORfEdkhIslEpIqIrAxWtjYQ5iwa\nESkhIhtEZLeIrBaRjHZ6LhH5y653j4jktIskFZFfROSoiMyTGPxCUoqUKSlfsTLr/lrD9evXOHTw\ngKMX2qjpm+zcYU2cmD9nJg2bNAOgZOky+Pn6cePG9RB1JUqUCF8/X8f6rz8voOobNXF3dydduvSU\nKlOOfXt3h2pDpkyZqdvAmmRUt0FjDh86EGK7t7c3C+fNoWPXnoz8ejBjJ82gdNly/LroJ0eeh35W\nAHa2O/d92bD3DDU9c5E2RWIK5czg6IX+8vdByhS03s+Xrt8jS3rrg83V1YXkSRI6ep2Z0yVn4dCW\ndBn6K2cuhT1hwsfPn4QJrEGWIrkzADjy/rLuEGUKvkqOzKl4LWNKdszoydGF75M5XXK2TuvOK6mT\nhqrvkj3Me9brFhv3naVo7pDD7PUr5GHv8Usk8UhAjkypaDvwZ5pUye+4z5kogRs+fo+e/sRFUfpX\nXsHLy+r5enl5OYYCAQ7s30/P7l34efEy0oTTU3FzcwtxEffo0SNaNf8fLVq1oXGTsO8vB1ehYiXO\nnDnN9esh38tffvEZAwcNYcL4sXTs3IWhw0Yw9KvH99p9/Xzx8PCI1rE+jUePHtGuVTOat2hNw8aP\njydd+le4bJ+3y15ejlsd8+b8SINGTRARcubMxWvZsnPi2NEQdSby8MDP9/Hf6Y7tW5k66QcK5cnB\ngP59WTB/Dl8O6B+qLZkyZ6ZBY6vuEqVK4+Liwo1g580Yw8jhQ/m4/wCGDx3M4KHf0KFTFyZNGOfI\n4+vnS6JYOG8vCvuzezpwxBjzXbD04H+QTYCD9uvlQEsRSSgi2YHcwA5gJ5BbRLKLSAKsSUXLI9t/\nbAbPNcCrInJcRCaISGW7oQuB94wxRbB6oz5hlK0KrH8y0b7qGAc0M8aUAGYAQ+3N84Af7HrLAUHj\nZ8WA97FmVeUAyofVWBHpFjTOfvOJQBaR69evcee2NdnEx8eHjevWkuv1PKRMmYp7d+9w6qR1hb1x\n3Vpef92ahZk5S1b+2WDdGzt+7Ah+fr6kDdbLBMiR63XO/3fOsZ45y6ts2midkgcPHrB753Zyv54n\nVHtq12vI5n82ALBl08ZQ9zQnjP2WLj164+7ujq+vDyKCi4sLPj5WcLp50xoGcnd3zgSXtCkSkyKp\nFYgTJXCjesmcHDt3nVv3fUmeJCG5slgf5NVKWekAv20+RpvaRQFoWjk/G/acASBF0kT8+k0bPp/8\nF1sPhjmIAcCxc9fIac/cvXTtHnmzpSOtPZM2aP+HTl/ltUYjydtiNHlbjObitbuU7TKZKzfvh6gr\nZdJEjvuqaVIkpmyhrBw5e82x3c3VhbebleW7+ZvxSOhG0J1TVxdxlMv9aloOn7n61OcwqurVb8jc\nObMAmDtnFvUbWPcv//vvP1o2b8r0mXPI/Xr4915zv56HM6et0QxjDD26diZP3ny898GH4ZY5dfKk\n437x3j178PPzCxGc/9m4gYwZM5Erd268vb0RFxdcXFzw9nk8BH/y+HHyFygYqm5nMsbwdo8u5MmT\nj7ff+yDEtjr1GjB/7mwA5s+dTd361v3LLK9mZcN6a2Ti6pUrnDx+jGzZc4QomypVKgICAvC1A+i0\nH+dy6MRZDhw7zZBhI2jZuh2DhgwL1Z56DRrxz4b1AJw8cZxHDx+SJm1ax/af5s2mZq26pE6dGh9v\nb1xcXHARF3zsESJjDFcvX3YMFz9vQSMdz7pEojzQDqj2xNdSRojIARHZjxU7PgAwxhwCFgGHgT+A\n3nYP1R94G1iNNelokZ03QrF2z9MYc19ESgAVsQ5oIVag8zLG7LTz3IWQ4+Uikhm4aYzxDlUp5AEK\nAn/aZVwBLxFJBmQ2xiyx6/UNVu8OY8wFe30fkA3YFEZ7pwBTAIoUKxHl2SNXL3vxbo/OBAQGEBgY\nSMMmzahZux4Ao8ZOpHO7Fri4uJAiZSpGj58CwMCh39Dn3Z5MmTAGEWHMhGmh3jhJkiQhW7YcnDl1\nkuw5c9Gpa0/e69WFSp5FMMbQsk0H8hcsDEDrZg35btwkMmTMxDsf9KVX1w5MmTCGJEmS8t24SY46\nL3tdYu/uXfTpZw2tderWi9pVy5IiRUpmzv8FgM0b11O9Vl2cJUOaZEz9tAmurtZU9MXrDvH7VuuC\novfI5fw0pAWBgYbb93zoPtyaqPHjb3uY8VlTDs5/l1v3fGg30Gpbj6alyZk5Nf07VKZ/h8oANPho\nDtduPwixz9+3HqdSsWys230arxv3+Hrmev4c34lH/gH8d/kO3YYtibDNxfNkokujkvQasZy82dIx\nrk8DAgMNLi7CqHmbOHrucfDs0bQ0c//Yh4/fIw6cukLihO7s/LEXq7ed4M596wO1UrFsfDEl/Ptf\n0dW+bSv+2bCe69evkzNbFj7/YhBvdepMn779aNuqObNmTidr1teY+5M1a3rYkMHcvHGD99/pBVg9\nzM3bQ9/XrlO3Hhs3rCdnrlxs2byZ+fPmULBgITxLWBcyg4Z8Te06dZk62XpPde3egyVLFjN/7mzc\n3dxJ5OHBnHkLHe9lYwzDvx7CnPkLAejcpRsd27fB39+fMeOtmaJXrlwhkYcHGTJkcNr5Ccu2LZtZ\nMH8uBQoWooKnNXHoi0FDqFm7Lh/2+YQObVsyZ9YMXs36Gj/OtSbi9O03gJ7dOlK2pPU3N2josBAB\nLkjVN2qwdcsmqlZ7I8I2vN2zK526dKd4iZK069CJ3t07U6ZEYdwTJGDitJmO8+bt7c38ObNZYs+o\n7v3uB7zZpD7uCRIw/UdrnsLePbspWdozzAlMz0NsPCTBGLMJaw7gk1ZFUGYojztYwdNXRVQuLBLd\nWYXOIiLNgN5AAmNM+Se2VcGe2CMinYHkxpjvg20fCNzHulKYYowp+0T5ZFhd+SxPpDvqtdfHA7uM\nMT9G1NYixUqYNRu2Pc1hOtWqFUvZv28P/T4fHGv77NTmTT4bNJScuaI/OzRb/SEx0KLoS5TAjdVj\n3qJq7+kEBj6f93uQ9KmS8OMXzaj7waxw89z6e1C422KTl5cXXTq257c//oy1fY4d/T3JkyfnrU6d\nn6r8Q//AyDPFsH179zBh3GimzJgda/v85KP3qVO/AVWqVo888xMqly/N3t27nBbuUryWz1T4NPz3\nd3Ss6uG5O7IJQ89LbE4YyiMiwccMi2J1kTOKSCk7TzIRefLSKdz7ncAxIJ09GQkRcReRAvZ3fi6I\nSGM7PaGIhP2N9zikboPGvJo1W6zt7+HDh9Su3/CpAueLxPehP1/NWEfmtGFPcIlNr76Sgn4/rH7e\nzYiSjBkz0rFz11CTYmJSypQpadu+Q6ztLyYULVacipWrhHhIQkzLV6DAUwXOmCDYz7d1wn8vstjs\n4ycFxolISqwnQ5wEugEz7XQPrPudjrEOsZ78kMsYczSM+jDGPLR7sGNFJAXW8YwGDmGNhU8WkcHA\nI+DNGDuyWNSmg9NnXIcrQYIENG/VLtb2F5P+2nnqeTcBgN1HX4zZkFHV7M3msbq/9m91jNX9xZR2\nsfh3CvBWp66xur/IRGWmbFwXm/c8d2NN3HnSdaxHKwW3HlgvIhV4/KXX4HUNDPZ6H1ApjDwngGpP\nJJ8m2MQjY8zbUWq8UkopFcyLcXc5HPYN4VCTeZRSSr2gojZTNs57oYOnUkqpuCcexE59tq1SSikV\nXdrzVEop5TQCUfk5sThPg6dSSimnigexU4dtlVJKqejSnqdSSimn0tm2SimlVDQE+xW3l5oO2yql\nlFLRpD1PpZRSTqWzbZVSSqloevlDZwTBU0TGAeH+fpMx5t0YaZFSSin1gouo5xn613GVUkqpSMTr\n2bbGmBC/ZioiiY0x3jHfJKWUUnGV9YSh592KmBfpbFsRKSsih4Gj9noREZkQ4y1TSimlXlBR+arK\naKAWcAPAGPMvYfx+plJKKRX0k2TOWF5kUZpta4w5/8SBBMRMc5RSSsV1L3jcc4qoBM/zIlIOMCLi\nDrwHHInZZimllFIvrqgEzx7AGCAzcAlYDfSOyUYppZSKu170IVdniDR4GmOuA21ioS1KKaXiOJ1t\naxORHCKyQkSuichVEVkmIjlio3FKKaXUiygqs23nA4uAjEAm4Gfgp5hslFJKqbgrPsy2jUrwTGyM\nmWOM8beXuUCimG6YUkqpuEmctLzIInq2bWr75e8i0g9YgPWs2xbAqlhom1JKKfVCimjC0G6sYBl0\nAdA92DYD9I+pRimllIqbROL5T5IZY7LHZkOUUkq9HOJB7IzaE4ZEpCCQn2D3Oo0xs2OqUUoppdSL\nLNLgKSJfAlWwgucqoA6wCdDgqZRSKpQXfaasM0Rltm0zoDpw2RjTESgCpIjRVimllIqzRJyzvMii\nEjx9jDGBgL+IJAeuAq/GbLOUUkqpF1dU7nnuEpGUwFSsGbj3ga0x2iqllFJxkiDxe7ZtEGNML/vl\nJBH5A0hujNkfs81SSikVJ8WBIVdniOghCcUj2maM2RMzTVJKKaVebBH1PL+NYJsBqjm5LS8sVxch\naaIofatHBXPr70HPuwlxUqpSbz/vJsRZl7eMed5NiHuM86uMD7NtI3pIQtXYbIhSSqmXQ1RmosZ1\n8eEYlVJKvWRE5FURWScih0XkkIi8Z6enFpE/ReSE/f9UdrqIyFgROSki+4PfmhSRDnb+EyLSISr7\n1+CplFLKaYRY+0kyf+AjY0x+oAzQW0TyA/2AtcaY3MBaex2sB/zktpduwERw/AjKl4AnUBr4Mijg\nRkSDp1JKKadyEecsETHGeAVNXDXG3AOOAJmBRsAsO9ssoLH9uhEw21i2ASlFJCNQC/jTGHPTGHML\n+BOoHekxRpbB7uq2FZEv7PWsIlI6snJKKaXip9gInsGJSDagGLAdeMUY42Vvugy8Yr/ODJwPVuyC\nnRZeesTHGIV2TQDKAq3s9XvAD1Eop5RSSj2LtCKyK9jS7ckMIpIUWAy8b4y5G3ybMcYQI/OJo/aE\nIU9jTHER2Ws35paIJIiJxiillIrbrOfSOu2rKteNMSXD35e4YwXOecaYX+3kKyKS0RjjZQ/LXrXT\nLxLy0bJZ7LSLWD9+Ejx9fWQNi0rP85GIuGJHbxFJBwRGoZxSSql4KDaGbcWK0NOBI8aY74JtWg4E\nzZjtACwLlt7evhVZBrhjD++uBmqKSCp7olBNOy1CUel5jgWWAOlFZCjWr6wMiEI5pZRSKqaUB9oB\nB0Rkn532KTAcWCQinYFzQHN72yqgLnAS8AY6AhhjborIV8BOO99gY8zNyHYelWfbzhOR3Vg/SyZA\nY2PMkSgenFJKqXgmNh4wZIzZhBWTwlI9jPwG6B1OXTOAGdHZf1R+DDsrVpReETzNGPNfdHaklFLq\n5Segv6pi+w3rfqcAiYDswDGgQAy2SymllHphRWXYtlDwdfuRRr3Cya6UUiqeiw9P34n2T4UYY/aI\niGdMNEYppVTcFw9GbaN0z/PDYKsuQHHgUoy1SCmllHrBRaXnmSzYa3+se6CLY6Y5Siml4jIR0QlD\n9sMRkhlj+sRSe5RSSsVx8SB2hn9fV0TcjDEBWF9EVUoppZQtop7nDqz7m/tEZDnwM/AgaGOw5wgq\npZT6f3v3HR9F0QZw/PekAEGq0rsU6RBC7whIkyK9g1TRFwsqHSkiSlFRLCAgUqSD9Galt1ClSldK\nKKEpBAJJ5v1jN0fCXSAJlxDI8/WTD3ezO7O74+09O7NzO8ohJjOiPKmic88zGXAZqM6933saQIOn\nUkqpSPQhCdazbN8F9nMvaIaLkylelFJKqSfBg4KnJ5AC188O1OCplFLKpUTQ8Hxg8AwwxnwYb3ui\nlFLqyReN6cSeBg96ilIiOHyllFIq5h7U8nSa0kUppZR6GEkEba8og2d0JgNVSimlIrJG2z7uvYh7\nieHh90oppZRbxXhWTAu16gAAIABJREFUFaWUUupBEkPLU4OnUkopt5JE8FsV7bZVSimlYkhbnkop\npdwmsQwY0uCplFLKfSRxPGFIu23jwOvdO5MrW0ZKlygaKX3E8KHkez4b5UuXoHzpEqxZtRKAubNn\nOtLKly5BymSe/Ll3j8uy27ZqzskTJwB4pX5dypXypZRvEd76Xw9CQ0Od1l+/bi1Z0qdxlP3JCOuh\nUZcuXeKlFytTukRRli1Z7Fi/ZdNXCDh3zvF+QN/3WfvH749WIdH0WtfO5MiSgZK+RSKlX7lyhZfr\nvESRgvl4uc5LXL16FYBlS5dQukQxypb0pWLZUmzauNFlubdu3eKl6lUJDQ1l7549VK1UHr/ihSld\nohjz5811mWfSdxMo5VuUsiV9qV61EocOHgRg86ZNlC5RjIplS3Hs6FEArl27Rv26tQgLC3Pkr1e7\npmM/3cnDQ9gyuy8Lv+zhSMuZ5TnWT3+f/UuGMGNkJ7y9PAGo6JeHzbP68p//lzSu6etYP0fmtGye\n1Zetc/qxc8FAujarFOX2Zo3pQq6szwHQok5J/OcNYPvc/iz5+g2eS/MMAEVfyMraae/hP28AC754\njZTPJHMqJ1vGNKye+Ba7Fg5k54KB/K91Nceyj95qxPa5/Zk8vL0jrVW90vRsc2+dwnmzMHFYu5hV\n1gOcOXOa+nVqUNavKOVKFmP8N+Mcyz4Y0IfSvoWpUKYEbVs25dq1awDMmzOLSmVLOv7SPuMd5Xna\noU0LTp20ztOXa1enVPFCjnyXLl50mefzMSMpUSQ/pYoX4rdf1gAQeOkSdWpUoXyp4ixfusSxbuvm\njSOdp4P692bd2vg5T5VFg2ccaNv+VRYvW+VyWc8332GL/262+O+mdt16ALRs3daRNumH6eTK9TzF\nivs65T148AChoaE8nzs3ANNnzWXrjj34795HYGAgPy2c73KbFSpWdpTff+BgAObPnU2Xbq+xbtM2\nvvn6SwBWLl9GMV9fMmfJ4sjb4403+XzMqNhXRgy07/gqS5avdkr/dPRIqlWvwf5DR6lWvQafjh4J\nwIvVa7B911627dzDhElTeKNHV5flTvthCo1eaYKnpyfJkyfn+x+ms2vvAZasWE2f995xfDlG1LJ1\nG3bs2ce2nXt49/0+9O39LgBffvEZi5atZPRnXzBp4gQARn78EX36DcDD497p1KZteyZO+PaR6+R+\nPdu8yF8nL0RKG/F2I76a+QdFGg3j6n+3eLVxeQBOB1yl+5AZzF29I9L6AZf+pVrHzyjXaiRV2o/h\n/U4vkTl9aqdtFcydCU8PD06dvYynpwdjejejTvcvKdPyE/YfPUuPllUBGD+4DYPGLaF0i49Z+sde\nenV0fr5KSGgY/T7/Cb+mI6ja4VNea1mFArkzkSpFMnwLZqdMy0+4czeUwnmzkCypNx0almPCvPWO\n/AeOnSNrxjRkz5T2kesQwMvTi48+GcO2Xfv4Ze0mJn83nsOHrAukF6vXZMuOvWzevpu8+fIx9lPr\n89aiVRs2btvJxm07+e77qeSM4jw9ZJ+nuZ7P7UibNGW6I2/6DBmc8hw+dJCFC+axdeefLFiygvfe\neZPQ0FAWzJ9Dp66v8dv6LYz/xjpPV61YRrHikc/T7q/35IvPRrulbtzBQ8QtfwmZBs84UKlyFdKm\nfTZWeRfMnU3TFi1dLps3eyb1GzR0vE+VKhUAISEh3L1zJ0Yj3Ly9vQkKCiI4OBhPD09CQkL45qsv\n6fVen0jr5ciZkytXLnPh/PlYHE3MVKpchWefda635cuW0K59RwDate/IsqVWSzlFihSOY75582aU\nxz9n9kwaNGwEQL4XXiBvvnwAZMmShfTpMxB46ZJTnvC6vb9sb29vbgUFcetWEN7e3pw4fpwzZ05T\npWq1SPlfbtCQeXNnx+TwHyprhjTUqVSYHxZtjpRetfQL/PTrbgBmLttGg2rFAfgn4Ar7j54jLCzy\nPA53Q0K5czcEgKRJvKP8kmpVrzTL1v4JWN1wIvCMTxIAUqbwIeDSdQDy5sjAxp3HAPh962FeqeEc\nUM4H/suew2cAuBEUzOGT58mSPg1hYcbRUk6eLAl3Q0J5p0MNxs9ZR0hIWKQyVq7bT/PaJaNTVQ+V\nKXNmfEv4WceSMiUv5C9AwLmzAFSvWQsvL+uOVqnS5Th39qxT/oXz5tC0WQuXZc+fM4t69Ru6XBaV\nlcuX0rRZC5ImTUquXM+TO08edu7YjreX9Xm7ExyMp6d1no7/Zhxvv9s7Uv4cOXJy5fKVeDlPHyb8\nnqc7/hKyBBU8RWSgiBwQkT9FZI+IlHVDmdVEpII79s8dvpvwDWVLFuf17p1ddustnD+P5i1bu8y7\nZctmfP0if3k0erkOz2fLSIqUKWncpJnLfNu3baFcKV8aN6jHwYMHAOsqesWypTSsV4v3+/Zn4oRv\nad22HcmTJ3fK71uiBFu2bIrpobrNxQsXyJw5MwCZMmXi4oV7La8lixdRvEgBmjR6mQkTpzjlvXPn\nDqdOniBnrlxOy/y3b+fO3TvkzpPH5XYnfPsNhfLnYWD/Pnw21urW692nP106dWDMqE/o8UZPhgwe\nyNBhHznlTZs2LcHBwVy+fDk2h+zSmN5NGfjl4kjB8Lk0z3D9v1uEhlqB5uyFq2TJ4NyKvF+2jGnY\nPrc/R1cN57OpvzoCYUTlfXOz+9BpAEJCwnj747n4zxvAiZ9HUDB3JqYutoL4oRMBNKhWDIAmL/mR\nLeODW4c5Mj+Lb/5s+O8/xY2gYNZsPMDWOf04H3idf2/conSRXI6gHdGug/9QoYTr/1eP4u+/T7Fv\n7x5Klnb+uvlx+g/UrFXHKf2nhfNp2qKVy/K2bt3sCMzh/tejK5XKlmT0Jx9hjPOkVAHnzpE1W3bH\n+yxZshFw7hzNWrZm5fKlvFK/Du/17sfkieNp2dr1eVrctwRbt252SldxI8EETxEpD9QH/IwxxYCa\nwOlHLNMLqAYkiODZtfvr7Dt0jC3+u8mYKTMD+r4Xabn/9m34JE9O4cJFXOa/EBBA+nTpI6UtWbGa\nY3+fIzg4mHUu7k36lvDj4NFTbN2xhx5v9KR1s8YApE6dmoVLlrNhiz++JfxYtWI5rzRpRs/Xu9G2\nVXO2bd3iKCNd+gyR7q88TiISqYXZ6JXG7N1/mHkLF/Ph0A+c1g8MDCR1mjRO6QEBAXTp1J7vJv0Q\nqbs1oh5v/I+Dfx3no49HMfJjK0AW9/Vl/aatrPn1D06dPEGmTJkxxtCuTUs6dWjHhQiBPb0b661u\n5SJcvPKfI5g9qjMXrlGm5ScUaTSMdg3KkOHZlE7rZEqXmsCr/wHg5eVBt2aVKdd6FLlrDWT/kbP0\n7lwLgNeGzqR7i8psmtmHFMmTcueu8733cM/4JGH2p13p/elC/rt5G4DPp/1KuVYj6ff5Iga/UZ/h\n45fzauPy/DiqM3271nbkvXj1P5fdy4/ixo0bdGjdgo9Hfx6ptwHg01Ef4+XlRYtWbSKl79i+jeTJ\nk1MoqvP0/HnSRThPJ02ZwWb/Paz6dS1bNm9kzqwfo71/qVOnZt6iZazdtI3ivn6sXrmcRo2b8tYb\nr9GhTQu2b7t3nqZPn57zAQnlPHXPX0KWYIInkBkINMYEAxhjAo0x50TklIiMFpF9IrJdRPICiEgu\nEfndbqX+JiI57PSpIjJBRLYB84AeQC+7JVtZRJqLyH4R2Ssi66PambiQMWNGPD098fDwoFPnbuzw\n94+0fMG8OTRv6fpqFiCZjw+3b992Tk+WjPoNGrJ82RKnZalSpSJFihQA1K5bj7shdwkMDIy0zqiP\nh9O73wDmz51N+QqVmPj9VD4ePsyxPPj2bXx8fGJ0rO6UIWNGAgICACvoubpnVKlyFU6ePOF0bD4u\n6uzff/+lScOXGfrhCMqWK/fQ7bdo2crRVRzOGMPIjz+i/8APGDF8GCM+GU3nrt349ut7A0/cWW/l\nfXNTv2pRDq8YxvSRnahW+gWmfNSBy9dukjqlD56e1qmcNWNazl10bkVGJeDSdQ4cC6Cin3OL7lbw\nHZIm8Qag+AvZADh5xqrfBb/solxx657ekVMXaPDGN1RsO5p5q3dy8oxzNzhYAXj2p92Yu2oHS37f\n67S8eP5siMCRUxdpUtOPdn2nkDtbevLksAJRsiTe3A6+G+1je5i7d+/SoU1zmrdqTcNXGkdaNnPG\nNNasWsGkH2Y43Q5YuGAuTZu7vrUC9nkafO8zlyVrVsDqHm7WojW7dvg75cmcJQtnz9y7MDp37kyk\ne5oAo0d+xHt9+rNw3hzKVajI+Ek/MHLEvVkjbwffxifZ4ztP7xE83PSXkCWk4PkzkF1EjojItyJS\nNcKy68aYosDXwBd22lfANLuVOhMYF2H9bEAFY0wTYAIw1hjja4zZAAwGahtjigMxuzHxiM7bAQBg\n2ZJFka5cw8LC+GnhfJo1jzp45i9QkBPHrXtLN27ccJQXEhLC6lUreSF/Aac8F86fd3QT7fDfTlhY\nGM8995xj+bGjRzl79ixVqlYjKCgIDw8PRIRbt29FWieqq+z48HL9hvw4YxoAP86YRv0G1v3L48eO\nOY5t965dBAcHRzo2sLpPQ0NDHQH0zp07tGzWmDbtOtCkqetubsAxkhZg1coV5M2bL9LymTOmU7tO\nPZ599lmCbtn15uFBUFAQYAXX8xfOu+wujo3BXy0lb50PKPDyEDr0+4G1/kfoPGg6AOt3HKFJzRIA\ntG1QluUuujwjypohDcmSWkExTUofKpTIw5FTziNA/zp5wRG4zl26ToHcmUiX1roQq1GuAH+dtO6v\npbfTRIR+3WozaYHrUc8ThrTlr5PnGfej61Ghg9+oz4ffrsDbyxNPT+uLM8yEkTyZdZ81X84MHDge\n4DJvTBlj6Pl6N17IX5Ceb/WKtOzXn1czbuynzJ6/2Kl7NCwsjMULFzwweObPX4CT9nkaEhLCZfuC\n7u7du6xZtYKChQo75an7cgMWLphHcHAwp06d5PixY5QsVcax/Pixo5w7e5bKVard+7yJcOtW5PO0\nYGHnslXcSDC/8zTG3BCRkkBl4EVgroj0sxfPjvDvWPt1eaCJ/XoGEHGo2XxjTFR9R5uAqSIyD/gp\nqv0Rke5Ad4DsOXLE6Fhebd+GDevXcjkwkBdyZ2fgB0Pp2KkLgwb05c+9exARcubMxbhvJjjybNyw\nnmzZsjtG0rpSp2491q9fy4s1anLz5k1aNG1EcHAwYWFhVKlaja7drZ8vTLZHgXbt3oNFPy1g8sQJ\neHl54ePjw9QZsyNdSQ8bMogh9j275i1b07p5Yz4bM4pBQ6yW5927dzl+/Bh+JUvFqA5io0O71mxY\nt5bAwEDy5MrGB4OH8WrnLrzfpx/tWrdg2g/fkyNHTn6cPQ+ARYsWMuvH6Xh7eZPMx4cZM+e6HDRU\ns2YtNm/aSPUaNVk4fx4bN6znyuXL/Dh9KgATv59KcV9fPhw6GL+SpajfoCHjv/2aP37/FW8vb9Kk\nTcukKdMc5QUFBTFj+lSWr/oZgLfeeZfGDeqRJEkSps6YBcCunTspU7acY+BJXBr45RJmjOzEkDfq\ns/ev00xdbHXllSyUg7mfdyNNquTUq1KUQT1epmSzEeR/PhMj322MwSAIX0z/jQPHnLv7Vm3YT5WS\n+fhj218EXLrOxxNX8cvkd7gbEso/AVfoPsTqfmxRpxSvtawCwJLf9zB9yVYAMqdPzbeD29D4zfFU\n8M1N2/pl2XfkLFvnWKf1kK+XsmajNcK1QbVi7Dr4j+Pe659/ncV/3gD2Hz3LviPWgJ2qpfKxesN+\nt9TZ1i2bmDvrRwoVKUqlstY4gsHDhlOrTj16v/s2d4KDeaW+da+zdJmyjP3KGjm9aeN6smbLFmkk\n7f1q1anHxvXrqFa9JsHBwTRpaPX4hIWGUvXFGnTsbI0KX7l8Gbt37WDg4GEULFSYxk2aUdavKF5e\nXnw6dhyenp6OMocP/YAPhg4HoFnzVrRt2YQvPhtN/w+GAtZ5evLEcUr4xf15+jBCwu9ydQdxdfM6\nIRCRZkBHoCjwojHmpIh4AwHGmHQiEghkNsbcvS99KrDcGLPALmcocMMY82mEsssCLwMdgJLGmAeO\n6vArWcps2OLc1RLfbt26Rb1a1fl17cZIJ1ZcWrpkEXt272KwfeLGhGcCGS63e9cuvvpyLFOmzYi3\nbb7X623qN2jIi9VjPi1u2tI942CPYi5ZUm/WTHyLFzt97jRiN74l8fbil8lvU73zWMfgKFfOb/4y\nHvfKtVu3btGgTg3W/L4h3s7TZUsWs3fPLgYN+fDhK9+nWsWy7N61w20na86CxUz/KUvdUtbrFZ7f\naYx5/FcELiSYblsRyS8iEfvGfIG/7dctI/wbfod8MxDex9kW2BBF0f8BjtEQIpLHGLPNGDMYuARk\njyJfguPj48PAD4a6HDofV0JCQnjrnfcevmICVsLPj6rVXnT5EIm4UrhwkVgFzoTkdvBdhk9YSdYM\nzgOu4lv2zGkZNG7pAwNnQuHj40P/QUM4dy7+ztPQ0BB6vv1uvG1PJaBuWyAF8JWIpAFCgGNY3ab1\ngbQi8icQDIT/juNN4AcR6Y0VBDtFUe4yYIGINLLz9LKDtAC/Ac4jFxKwmrVqP3wlN2rStHm8bi+u\ndOzUOV6317lrt3jdXlz5dcuhx70LABz/5xLH/3E9ECkhqvFS/J6nr0TxM7XHJaE/4MAdEkzwNMbs\nxMVPSux7WGOMMX3vW/9voLqLcl697/0RoFiEpKhaqEoppR5RYrnnmWC6bZVSSqknRYJpeUbFGJPr\nce+DUkqp6NNuW6WUUiqGEkHs1G5bpZRSKqa05amUUspthMTRKtPgqZRSyn0El0/6etokhgsEpZRS\nTyERmSIiF0Vkf4S0oSJy1p4MZI+I1IuwrL+IHBORv0SkdoT0OnbasQiPhX0gDZ5KKaXcStz0Fw1T\nAecJV+9NBuJrjFkJICKFsJ5KV9jO862IeIqIJ/ANUBcoBLS2130g7bZVSinlNkL8/VTFGLNeRHJF\nc/VGwBx72suTInIMCJ+65pgx5gSAiMyx1z34oMK05amUUiqhSiciOyL8dY9mvp72XM9TRCStnZYV\niDib/Bk7Lar0B9LgqZRSyq3c2G0baIwpFeFvYjQ2Px7IgzW5SADwmbuOKyLttlVKKeVWj3OwrTHm\nwr39kEnAcvvtWSLPopXNTuMB6VHSlqdSSqmnhohkjvC2MRA+Encp0EpEkorI80A+YDvgD+QTkedF\nJAnWoKKHTkiqLU+llFJuJPH2O08RmQ1Uw7o3egYYAlQTEV/AAKeA1wCMMQdEZB7WQKAQ4H/GmFC7\nnJ7AGsATmGKMOfCwbWvwVEop5Tbx+YQhY0xrF8nfP2D9EcAIF+krgZUx2bYGT6WUUm6lTxhSSiml\nlBNteSqllHKrp7/dqcFTKaWUO+mD4ZVSSinlirY8lVJKuY3O56mUUkrFgnbbKqWUUsqJtjyVUkq5\n1dPf7tTgqZRSys0SQa+tdtsqpZRSMaUtT6WUUm5jjbZ9+pueGjyVUkq5lXbbKqWUUsqJtjyVUkq5\nkSDabauUUkrFjHbbKqWUUsqJtjyVUkq5jY62VUoppWJKEke3rQbPaAoLM497F544WmexE7jtq8e9\nC0+sdBV6Pe5deOIE/3X6ce/CE0mDp1JKKbfSlqdSSikVQ4nhpyo62lYppZSKIW15KqWUchsBPJ7+\nhqcGT6WUUu6l3bZKKaWUcqItT6WUUm6lo22VUkqpGNJuW6WUUko50ZanUkopt9HRtkoppVSMJY75\nPLXbVimllIohbXkqpZRyH51VRSmllIq5RBA7tdtWKaWUiilteSqllHIba7Tt09/21OCplFLKrZ7+\n0KndtkoppVSMactTKaWUeyWCpqcGT6WUUm6lD0lQSimlEigRmSIiF0Vkf4S0Z0XkFxE5av+b1k4X\nERknIsdE5E8R8YuQp6O9/lER6RidbWvwVEop5VYi7vmLhqlAnfvS+gG/GWPyAb/Z7wHqAvnsv+7A\neGtf5VlgCFAWKAMMCQ+4D6LBUymllFuJm/4exhizHrhyX3IjYJr9ehrwSoT06cayFUgjIpmB2sAv\nxpgrxpirwC84B2Qnes9TKaVUQpVORHZEeD/RGDPxIXkyGmMC7NfngYz266zA6QjrnbHTokp/IA2e\nSiml3Mt944UCjTGlYpvZGGNExLhtbyLQblullFJuY3W5uue/WLpgd8di/3vRTj8LZI+wXjY7Lar0\nB9LgqZRS6mmyFAgfMdsRWBIhvYM96rYccN3u3l0D1BKRtPZAoVp22gNpt61SSin3iccpyURkNlAN\n697oGaxRsyOBeSLSBfgbaGGvvhKoBxwDgoBOAMaYKyIyHPC31/vQGHP/ICQnGjyVUkq5VXw9IsEY\n0zqKRTVcrGuA/0VRzhRgSky2rd22SimlVAxpy1MppZR7Pf1P59OWZ1x447Uu5M6RibIli0VK/3Pv\nHqpXqUDFsn5UrViGHf7bAbh69SptWjShfGlfqlUqx8ED+10VizGG+nVq8u+//zrSQkNDqVSuJM2b\nNHCZ5+svx1K6RBHKl/alQd2X+OfvvwE4euQvqlQoTfnSvmzbugWAkJAQGtarRVBQkCP/q+1bc+zY\n0dhXRgxEVW/7/txLjaoVKVeqOC2aNox0/ACn//mHzOlSMW7sZy7LjU29fT9pAuVKFadiWT9qVa/C\n4UMHAdi6eRPlS/tStWIZR71cu3aNRvVrExYW5sjfsF4trl69GvNKiIXXu3cmV7aMlC5RNFL6iOFD\nyfd8NsqXLkH50iVYs2qlY9n+fX9SvUoFSvkWoYxfMW7fvu2y7LatmnPyxAkAXqlfl3KlfCnlW4S3\n/teD0NBQp/XXr1tLlvRpHNv8ZMSHAFy6dImXXqxM6RJFWbZksWP9lk1fIeDcOcf7AX3fZ+0fv8e+\nMu6TNIkXG6b1Ytus3uyc25dB3e/99r1qqXxs/vE9dszty6ShbfD0vPd1WLlkXrbOtPL8/F1PR/qE\nwa35++fh7Jjb94Hb7dm6Km1eLg1AsReysu6Hd9g6szcbp79LqcI5AEj1TDIWfN7VsW/tG5RxWVaJ\nAtnwn9OH/YsG8tn7TRzpH73ZgO2z+zB5WFtHWqu6JenZuqrjfeE8mZk4pE10qsqN3DXWNmFHYA2e\ncaBt+478tGSlU/oHA/vSb+AHbNq2iwEfDGXwQOupUZ+N/oSixX3Z4r+Hid9Ppe/7vVyWu2b1SooU\nLUaqVKkcaeO/HscL+QtEuS/FfH1Zt2k7W/z30KhxEwYPtE76KZMnMmrMWBYsWs5XX1hBZ/LECbRs\n3YbkyZM78nft3oMvPx8T80qIhajqrefr3Rn20cds3bGXBg1f4cuxn0ZaPqDve7xUK+oHgsSm3pq3\nbMPWHXvZtG0X77z7Pv37vgfAV19+zoJFyxk5+nOmTPoOgDEjR/B+n/54eNw7nVq2acvkieOjd+CP\nqG37V1m8bJXLZT3ffIct/rvZ4r+b2nXrAdZFUpdX2/Pl1+PZsWc/q375A29vb6e8Bw8eIDQ0lOdz\n5wZg+qy5bN2xB//d+wgMDOSnhfNdbrNCxcqObfYfOBiA+XNn06Xba6zbtI1vvv4SgJXLl1HM15fM\nWbI48vZ4400+HzMq9pVxn+A7IdTp8Q1l24yhbJsx1KpQkDJFciIiTB7ahg4DplOq5Sj+CbhKu/pW\nsEudwocv+zaj+buTKNlyFG37TXWUN2PZNhq9+d0Dt+np6UGHhmWZu3onACPeasCISWso13YMw79b\nxYi3GgLwWotKHD55gbJtxlD7ta8Z+U4jvL08ncob1785//toLkUajyBP9vTUqlCQVM8kw7dANsq0\nHs2duyEUzpOZZEm96dCgLBPmbXDkPXA8gKwZUpM9Y5pHrUp1Hw2ecaBipSqkffZZp3QR4T+79fPv\n9etkypwZgMOHD1K16osAvJC/AH//fYqLFy445Z83ZxYvN2joeH/2zBnWrF5Jx05dotyXKlVfdATD\n0mXKcfas9fMlb29vgm4FEXQrCC9vb65du8aqlcto3bZDpPwVKlZm7e+/ERISEpMqiJWo6u34sSNU\nrFQFgBerv8TSxT85li1fupicuZ6nQKHCUZYbm3qLGGhv3ryJ2MMHI9abt7c3J04c5+yZ01SuUi1S\n/novN2TBvDkPPmA3qVS5CmnTOtdbVH775WeKFC1G0WLFAXjuuefw9HT+0p43eyb1I9RbeJ2EhIRw\n984dR51Eh7e3N0FBQQQHB+Pp4UlISAjffPUlvd7rE2m9HDlzcuXKZS6cPx/tsh/m5q071j54eeLl\n5YEx8Fzq5NwJCeXYP5cA+H3bX7xS3aqPlnX8WPLHn5y+cA2AS1dvOMratPsEV/4N4kGqlcrHnsNn\nCA21eiKMsVqZYAXmgEvXHekpkicF4JnkSbn6bxAhoWGRysr0XCpSPpOM7futHqNZK/1pUK0oYcY4\nAm3yZEm4GxLKO+1eZPzcDU5lrNxwgOa1/YhP8fhs28cmQQZPERkrIu9EeL9GRCZHeP+ZiLwbzbKm\nikgzF+nVRGS5e/Y4ekaNGcsHA/pSMG9OBvXvw9APPwagaNHiLF2yCIAd/ts5/c/fnD17xin/ti2b\n8S1R0vG+X+9efDhiZKQWz4NMnzqFl2pbLbRur73BZ6NH0qNrJ97v05/Rn3zk1HoC8PDwIHeePOz7\nc2+sjtkdChQszIpl1k+1Fv+0gLNnrCdp3bhxg7GfjaGf3bqJSmzrbeKEbylWKB+DB/Zj9GdWa+nd\n3v14rcurfD5mFN17/I8Phwxi0NDhTnnTpk1LcHAwly9fjsmhut13E76hbMnivN69s6Mb+djRI4gI\njV6uQ8WyJRn76WiXebds2YyvX8lIaY1ersPz2TKSImVKGjdxOq0A2L5tC+VK+dK4QT0OHjwAQItW\nbVixbCkN69Xi/b79mTjhW1q3bReplyOcb4kSbNmy6VEOOxIPD2HrzN7888tH/L7tCP4H/ibw2k28\nPD3wK2j9Nr5ue3o2AAAYM0lEQVRxjeJks1tn+XJkIE1KH9Z815NNM95zdL9GV/niz7P78L2nvfX+\nbBEfv92Qo8uH8MnbDRn8tfW1M2HeBgo8n5ETq4exY05f3v90EdaA0HuyZEjNWTuIA5y9cI0s6VNz\nIyiYNZsOsnVmb84H/su/N25TukhOlq3b57Q/uw6dpoJv7hgdw6Nw13NtE3jsTJjBE9gEVAAQEQ8g\nHRCxaVEB2PywQkTE+XL6MZo8cQKfjP6MQ8f+5pPRn9Hz9W4A9Hq/L9evX6NiWT++G/81xYqXcNkS\nuHr1CilTpgRg1crlpMuQgRL3fblFZc7sH9m9aydv93ofgOw5crDy59/5bd0mfJIn59zZM+TPX5Bu\nnTvwartWHD16xJE3XfoMnA84F1XRce7b7yYzaeJ4qlQozX83/sM7SRIAPvloGP97821SpEjxwPyx\nrbfuPd7gz4NHGfbRJ4wZOQKAYsV9+X39Zlas+Y1Tp06QKVNmjDG82q4VXTu1j9RjkP4x11vX7q+z\n79AxtvjvJmOmzAywu55DQkLYsmkj30/7kV/+2MCypYv54/ffnPJfCAggfbr0kdKWrFjNsb/PERwc\nzDoX9yZ9S/hx8Ogptu7YQ483etK6WWMAUqdOzcIly9mwxR/fEn6sWrGcV5o0o+fr3WjbqrnjvjtY\nn7eI90EfVViYoVzbMeStN5RShXNQKE8mADoMmM7od19hw7Re/BcUTGioFbi8vKyg2vjtiTTsOYH+\nXWqRN0f6B20ikkzpUhEYobXavVlF+ny+iHz1h9Hn88WM/6AVAC+VL8CfR86Su84QyrYZw9g+TUn5\nTNJob+fz6b9Tru0Y+n2xhMGv12X4hFW82qgcP37Skb5dXnKsd/HKf2ROnzra5bpFIoieCTV4bgbK\n268LA/uB/+wnQCQFCgKpRWS3iOyz53RLCiAip0RklIjsAppHLFRE6ojIYXtZE+LZ7JnTafiKtdnG\nTZuzc4c1YChVqlSMnziFTdt2MfH7aVwOvESu552vFL28vByDUrZt2cyq5csokj83nTq0Yf3aP+ja\nqb3L7f7x+698OuoT5i5YTNKkzifncLv1NOHbr+jYqQsfjhjFSHugB0Dw7dsk8/F55OOPrRfyF2DJ\n8jWs3+xPsxateP75PIDVSh88sB9F8udm/Ndf8umYT/hu/DdO+WNbb+GatWjlaPmGM8YweuQI+vQf\nxMgRH/LhiFG82rkrE779yrFOcPDjrbeMGTPi6emJh4cHnTp3Y4e/9RvwLNmyUbFyFdKlS0fy5Mmp\nVacue3fvcsqfzMfH5UCiZMmSUb9BQ5bfVydgfZbDL2Zq163H3ZC7BAYGRlpn1MfD6d1vAPPnzqZ8\nhUpM/H4qHw8f5lgefPs2PnFQb9dv3GLdjmPUKl8QgG37TlGz21dU7jiWjbuOc+wf6yluZy9c45ct\nhwm6fYfL12+ycfdxiuXL8qCiI7kdfJekSe7dQ25bvzSLf/8TgIW/7qFU4ZwAtG9QhiV2+okzgZw6\nd5n8uTJGKuvcxetkjXC/MmvGNJyzu33DFc+fFRHhyN8XaVLTl3b9p5E7azryZE8HQLIk3twOvhvt\n/VfRkyCDpzHmHBAiIjmwWplbgG1YAbUUcBSYDLQ0xhTF+snN6xGKuGyM8TPGOG46iUgyYBLQACgJ\nZIqPY4koU+YsbNywDoB1a38nT958gDVa884d677MtB8mU6FS5Uj33MLlzZefkyetkY9Dh3/M4eP/\nsP+vE/wwfRZVqr3I5B9mOOXZu2c3b/d8nTkLFpM+Qwan5Rs3rCNT5izkzZuPoKAgPMQDDw8PbkUY\ncXvs2FEKFSry6BUQS5cuWl9qYWFhjBk5gi7dugOw5rd17P/rBPv/OsHrPd/m/d79ee11599Ax6be\nIo4wXrNqheP/VbhZM6dTq3Y9nn32WW4FBeHh4YGHeDhGKhtjuHD+PDlz5nJLHcTG+YAAx+tlSxZR\nqLD1/7DmS7U5sH8fQUFBhISEsHH9egoULOSUP3+Bgpw4fgywusjDywsJCWH1qpUuB1xdOH/e0fW4\nw387YWFhPPfcc47lx44e5ezZs1SpWs36vHl4ICLcun0r0jrh+/qo0qV5htQprECcLKk3Ncq+wF+n\nrN6B9GmtIJ/E25P3OtZg0kKrM2vZuv1U8M2Np6cHPkm9KV0kJ4dPOY9BiMrhUxccgQsg4NK/VC6Z\nF4BqpfNx7LR1n/X0+WtUK/MCABmeTcELOTNw8kzkbv7zl//lv5u3KVPECrht6pVm+X1ds4N71OPD\n8Svx9vLE08NqroUZQ/JkVg9NvpzpOXA8gPiUGEbbJuTfeW7GCpwVgM+xpoipAFzHmjLmljEmvG9x\nGtaTI76w3891UV4B4KQx5iiAiPyINSGqSyLSPXx59uw5YrTjnTq0YeOGdVwODKRAnhwM+GAIHV7t\nwlfffEff3r0ICQkhadJkfPn1BAD+OnyIHt06ISIULFiIrydMdllu7br12Lh+LXny5H3g9j/6cAh+\nfiWpV78hHwzoy82bN+jYtiUA2bJnZ+4Cq8VgjGHMyBH8MMO6xujUpRtdO7UnJCSEsV9aLbiLFy6Q\nLJkPGTPF/bVGVPU2f94cJn33LQANGzWmXYdOMSo3NvU2cfw3rP3jN7y9vUmTJi0TJv3gWC8oKIhZ\nM6azePlqAHq+1YtmjeuTJEkSvp/6IwC7d+2kdJmyeHnF/Sn2avs2bFi/lsuBgbyQOzsDPxhKx05d\nGDSgL3/u3YOIkDNnLsZ9Y33e0qZNy5tv96JKhTKICLXr1KVOvZedyq1Ttx7r16/lxRo1uXnzJi2a\nNiI4OJiwsDCqVK1G1+49AOt2BFgjsxf9tIDJEyfg5eWFj48PU2fMjjSwaNiQQQwZ9hEAzVu2pnXz\nxnw2ZhSDhlgtz7t373L8+DH8SsZ6Io1IMqVLxaRhbfH08MDDQ1j4yx5WbbR+dtSrfXXqVi6Mh4cw\nacEm1u2wLpj+OnWBX7Ycwn92H8KMYerirRw8bg1gmjaiA5VL5iFdmhQcWzGU4RNXMW3Jtkjb/HnT\nIb7/sJ3j/f8+msOY95vg5elB8J0Qeo6wvp5GTl7DxKFt8J/TBxFh4FfLuHz9JgBbZ/amXFtrlPvb\nIxcwcWgbfJJ68/PmQ6zZdMhRdoOqRdl16DQBgdZAxD+PnLV+1nL0HPuOWl3fVUvlY7V9zPEloQ/2\ncQe5/wZ1QiEib2AFvEpAaSA1MB/4F1gLNDXGVLHXrQH8zxjTREROAaWMMYH2sqnAcqznGY6LkKch\n0N0YU/9h++JXspRZt2m7W48vNs4HBPBa144sWfFzvG3z63FfkCpVSjq8GvXI1ITucdRbn/feoV79\nBlR70ekpYQ/l4ZEwvnlu3bpFvVrV+XXtRpf34OPC0iWL2LN7F4NdDMKKjnQVXP/MK77NHdOZAeOW\ncvx04MNXjkNJvD35ZeKbVO86zjH6937Bh2YTdvOC2z50hYv5mTkr17ulrGLZU+58lCnJ4lKC7La1\nbQbqA1eMMaH2g3rTYHXdLgRyiUh4U6I9sO4h5R228+Sx30f1TMQEK1PmzHTs1NXpIQFxKU2a1LRp\n1/HhKyZgj6PeChUuHKvAmZD4+Pgw8IOhnDv70NmZ3CYkJIS33nkv3rYXVwZ9vZxM6ZxvvcS37JnS\nMujr5VEGzriSCMYLJehu231Yo2xn3ZeWwhhzRkQ6AfNFxAvrafgTHlSYMea23RW7QkSCgA1AyrjZ\n9bjTpFmLh6/kRjHtIk2o4rveXu3cLV63F1dq1qodr9tr0rT5w1d6Ahz9+yJH/7748BXj2PHTgfHf\n+n0SIp8bJNjgaYwJBVLdl/ZqhNe/ASVc5Mv1gDyrsbqClVJKqVhLsMFTKaXUkymhj5R1Bw2eSiml\n3EZIHKNtE/KAIaWUUipB0panUkopt0oEDU8NnkoppdwsEURP7bZVSimlYkhbnkoppdxKR9sqpZRS\nMaSjbZVSSinlRFueSiml3CoRNDw1eCqllHKzRBA9tdtWKaWUiiFteSqllHIba1KVp7/pqcFTKaWU\n+4iOtlVKKaWUC9ryVEop5VaJoOGpwVMppZSbJYLoqd22SimlVAxpy1MppZQbiY62VUoppWJKR9sq\npZRSyom2PJVSSrmNkCjGC2nwVEop5WaJIHpqt61SSikVQ9ryVEop5VaJYbSttjyVUkq5lYh7/h6+\nHTklIvtEZI+I7LDTnhWRX0TkqP1vWjtdRGSciBwTkT9FxO9RjlGDp1JKqSfZi8YYX2NMKft9P+A3\nY0w+4Df7PUBdIJ/91x0Y/ygb1eCplFLKrcRNf7HUCJhmv54GvBIhfbqxbAXSiEjm2G5Eg6dSSin3\ncVOXbTQftGCAn0Vkp4h0t9MyGmMC7NfngYz266zA6Qh5z9hpsaIDhpRSSiVU6cLvZdomGmMmRnhf\nyRhzVkQyAL+IyOGImY0xRkRMXOyYBk+llFJu5rbRtoER7mU6Mcactf+9KCKLgDLABRHJbIwJsLtl\nL9qrnwWyR8iezU6LFe22VUop5TZC/HTbisgzIpIy/DVQC9gPLAU62qt1BJbYr5cCHexRt+WA6xG6\nd2NMW55KKaWeRBmBRWJFWS9gljFmtYj4A/NEpAvwN9DCXn8lUA84BgQBnR5l4xo8lVJKuVV8PCLB\nGHMCKO4i/TJQw0W6Af7nru1r8IyG3bt2Bqby8fz7ce9HFNIBgY97J55AWm+xo/UWOwm53nK6u8DE\nMCWZBs9oMMakf9z7EBUR2fGgG+rKNa232NF6ix2tt6ePBk+llFJulRiebavBUymllHs9/bFTf6ry\nFJj48FWUC1pvsaP1Fjtab08ZbXk+4e572oaKJq232NF6i53EVm+JoOGpwVMppZT7xOC5tE807baN\nJyISas85t1dEdolIhViW00NEOrh7/54EIjJQRA7Yc/HtEZGyUaxXSkTGRXjvLSIn7Tx7ROS8iJyN\n8D5JNLdfTUSWu+t4Hqfo1mUMy6wW28/1k0JExorIOxHerxGRyRHefyYi70azrKki0sxF+lPzOXua\nacsz/twyxvgCiEht4BOgakwLMcZMcPeOPQlEpDxQH/AzxgSLSDrAZdAzxuwAIj5MuhKw3Bjzpl3W\nUOCGMebTuN3rhCkmdRmDMr2AasANYPMj72TCtQnriTVfiIgH1u83U0VYXgHo9bBCRMQzbnYvYUgM\no2215fl4pAKugvNVpoh8LSKv2q9HishBu3XwqZ02VETet1+vFZFRIrJdRI6ISGU73VNExoiIv533\nNTs9s4ist1sa+0Wksr3uVPv9PhF56In/mGTGekh0MIAxJtAYc05ESovIZrtFv11EUrq4cq8DrHJV\nqIiUFJF19pRGa8Ln9xORvCLya4Segjx2lhQiskBEDovITJEnsoMqqro8JSKj7c/BdhHJCyAiuUTk\nd/uz9JuI5LDTp4rIBBHZBswDegC97M9XZRFpbn+u9orI+sd1sG62GShvvy6M9SzV/0QkrYgkBQoC\nqUVkt12PU+x07PodJSK7gOYRCxWROvZnahfQJP4OJ4485gk944O2POOPj4jsAZJhfXlVf9DKIvIc\n0BgoYE+rkyaKVb2MMWVEpB4wBKgJdMF66HFp+8TdJCI/Y52Ua4wxI+wr3+SAL5DVGFPE3m5U23nc\nfgYGi8gR4FdgLrDF/relMcZfRFIBt1zkfREYdn+iiHgDXwGNjDGXRKQlMALoDMwERhpjFolIMqwL\nzexACawvzXNYrZCKwEa3Hmncc6pLY8w6e9l1Y0xRsW4NfIHVQv0KmGaMmSYinYFx3JtgOBtQwRgT\nen+LXkT2AbXtKaMS6ucqRuyLjBD7AqIC1mcwK1ZAvQ4cBSYDNYwxR0RkOvA6Vl0CXDbG+IEVMO1/\nkwGTsL4TjmF9plUCpy3P+HPLGONrjCmA1RKa/pBWy3XgNvC9iDTBepCxKz/Z/+4Ectmva2HNHrAH\n2AY8B+QD/IFO9pdcUWPMf8AJILeIfGWfzP/G9gDjkjHmBlAS6A5cwvqCeQ0IMMb42+v8a4wJiZhP\nRLICV4wxruovP1AEax7APcAgIJtYMzVkNcYsssu9HSH/dmPMGWNMGLCHe3X+xHBVl+G9HcDsCP+G\nt7DKA7Ps1zOwusHDzTfGhEaxqU3AVBHpBjxN3ZSbsQJnePDcEuH9GeCkMeaIve40oEqEvK4CYwE7\nz1H7+as/xtWOx5dE0PDUlufjYIzZYt9nSg+EEPkiJpm9ToiIlMF6wHEzoCeuW6vB9r+h3Pv/KcCb\nxpg1968sIlWAl7G+1D43xkwXkeJAbaxutxZYLa8Ex/6SXgustVs10XnIcx3AqR5sAhwwxpSPlGhP\ncxSF4AivI9b5E8VFXYZP4RRx4uDoTCJ88wHb6CHWQKSXgZ0iUtJ+aPeTbhNWoCyK1W17GngP68Jz\nLdD0AXmjrK+nyRN5MyOGtOX5GIhIAawr8ctYU+YUEpGkdtdWDXudFEBqY8xKrAEITrMHPMAa4HW7\nWxIReUGsue9yAheMMZOwupb87CDuYYxZiNXy8nPPUbqXiOQXkXwRknyBQ0BmESltr5NSrIErEUV5\nvxP4C0gv1gCa8FG5he0W+RkRecVOTyoiyd15PI9TFHUZPvFBywj/brFfbwZa2a/bAhuiKPo/wHHh\nISJ5jDHbjDGDsVq42aPI96TZjNWdfcUYE2qMuQKkwWqhLwRyhd8vBtoD61wX43DYzhN+X711HOxz\nPBK3/ZeQPZFXzU+o8HueYLV4OtpX/6dFZB7WFexJYLe9TkpgiX0/RIBoDX+3TcbqTtxldw1fwrpH\nVQ3oLSJ3sUZFdsC6X/ODWCMHAfrH7vDiXArgK/sCIwTr3lB34Ac73QfrfmfN8Az2fd28xpjDrgo0\nxtwR66cC40QkNdb58AVwAOtL7zsR+RC4y30DPJ5wUdVlfSCtiPyJ1cIO/xJ/E+sz0hvrsxTVPIjL\ngAUi0sjO08sO0gL8BuyNo+OJb/uwRtnOui8thTHmjIh0AubbF3L+wANHyBtjbotId2CFiARhXZw8\nqPdDJQBidbEr9fQRkUpAO2NMj8e9L08CETkFlDLGJNSps9QToIRfKfP7xm1uKevZZ7x2JtTZaLTl\nqZ5axpiNPHkjYZVSTwANnkopAIwxuR73Pij1pNDgqZRSyq0Sw2hbDZ5KKaXcKqGPlHUH/amKUkop\nFUMaPFWiJ/dmvNkvIvMf5TedEmGmDBGZLCKFHrBurGYhsZ+Rmi666fetcyOG23I8S1mpaJF705I9\n6l9CpsFTqXuPTiwC3MF60pKDiwcvRIsxpqsx5uADVqmG9aQapZ4a7no0XwKPnRo8lbrPBiCv3Src\nICJLgYMS9Uw1ItZMOH+JyK9AhvCCxJr1ppT9uo5Ys7PsFWtmklw4z0KSXkQW2tvwF5GKdt7nRORn\nsebfnEw0vldEZLFYM8UcsH+AH3HZWDv9NxFJb6flEZHVdp4N9lOwlFJR0AFDStnsFmZdYLWd5AcU\nMcactAOQq5lqSmA9YL4QkBE4CEy5r9z0WLNmVLHLetYYc0VEJhB5FpJZwFhjzEaxZu1YgzXF1RBg\nozHmQxF5GWvWnIfpbG/DB/AXkYX2c2WfAXYYY3qJyGC77J7ARKCHMeaoWM+j/ZaHzPyjVJQSerPR\nDTR4KhX50YkbgO+xulO3G2NO2um1gGLh9zOB1Fgz1VQBZtuPWjwnIr+7KL8csD68LPtZqK7UxHrO\ncfj7VGI947gK9hyPxpgVInI1Gsf0log0tl9nt/f1MhDGvZk9fgR+srdRAeuRcuH5k0ZjG0q5lBhG\n22rwVMq+5xkxwQ4iEWfAcDlTjVjzqLqLB1DOGHPbxb5Em4hUwwrE5Y0xQSKyFnu2HheMvd1r99eB\nUipqes9TqehxOVMNsB5oad8TzYw18fb9tgJVROR5O++zdnqkWUiwJql+M/yNiIQHs/VAGzutLpD2\nIfuaGrhqB84CWC3fcB5YU9xhl7nRGPMvcFJEmtvbELGmqVMqVnS0rVIq3GSs+5m7RGQ/8B1Wz80i\n4Ki9bDr3pvFyMMZcwpq15CcR2cu9btNlQOPwAUPAW0Ape0DSQe6N+h2GFXwPYHXf/vOQfV0NeInI\nIWAkVvAOdxMoYx9DdeBDO70t0MXevwNAo2jUiVIuJYbRtjqrilJKKbfxK1nKbNzi75aynknqobOq\nKKWUSiQSerPRDTR4KqWUcqvEMNpW73kqpZRSMaQtT6WUUm4jJPyRsu6gA4aUUkq5jYisBh44QUEM\nBBpj6ripLLfS4KmUUkrFkN7zVEoppWJIg6dSSikVQxo8lVJKqRjS4KmUUkrFkAZPpZRSKob+D6tA\nEeG+uxUxAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 504x504 with 2 Axes>"]},"metadata":{"tags":[]}},{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","           0       0.83      0.82      0.82      4500\n","           1       0.82      0.84      0.83      4500\n","           2       0.92      0.89      0.91      4500\n","           3       0.86      0.88      0.87      4500\n","\n","    accuracy                           0.86     18000\n","   macro avg       0.86      0.86      0.86     18000\n","weighted avg       0.86      0.86      0.86     18000\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ir3b-qTroDMr","colab_type":"text"},"source":["# inference.py"]},{"cell_type":"markdown","metadata":{"id":"0VjPzdW0oFOE","colab_type":"text"},"source":["## Load model"]},{"cell_type":"code","metadata":{"id":"RBS-k5PpnzDK","colab_type":"code","outputId":"994b4c7e-5e2d-458f-c721-caa7525b0fa7","executionInfo":{"status":"ok","timestamp":1584478731033,"user_tz":420,"elapsed":1292,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Load model\n","model = TextCNN(embedding_dim=EMBEDDING_DIM,\n","                vocab_size=vocab_size,\n","                num_filters=NUM_FILTERS,\n","                filter_sizes=FILTER_SIZES,\n","                hidden_dim=HIDDEN_DIM,\n","                dropout_p=DROPOUT_P,\n","                num_classes=len(classes),\n","                pretrained_embeddings=embedding_matrix,\n","                freeze_embeddings=False).to(DEVICE)\n","model.load_state_dict(torch.load(MODEL_PATH))\n","model.eval()"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["TextCNN(\n","  (embeddings): Embedding(35635, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=128, bias=True)\n","  (fc2): Linear(in_features=128, out_features=4, bias=True)\n",")"]},"metadata":{"tags":[]},"execution_count":70}]},{"cell_type":"markdown","metadata":{"id":"H8yG34NwoMgF","colab_type":"text"},"source":["## Inference"]},{"cell_type":"code","metadata":{"id":"JtYSTdIKoNp-","colab_type":"code","colab":{}},"source":["import collections"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Q_ggwcbwobjG","colab_type":"code","colab":{}},"source":["def get_probability_distribution(y_prob, classes):\n","    results = {}\n","    for i, class_ in enumerate(classes):\n","        results[class_] = np.float64(y_prob[i])\n","    sorted_results = {k: v for k, v in sorted(\n","        results.items(), key=lambda item: item[1], reverse=True)}\n","    return sorted_results"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"79RHYZtFCtEz","colab_type":"code","colab":{}},"source":["def get_top_n_grams(tokens, conv_outputs, filter_sizes):\n","    # Process conv outputs for each unique filter size\n","    n_grams = {}\n","    for i, filter_size in enumerate(filter_sizes):\n","        \n","        # Identify most important n-gram (excluding last token)\n","        popular_indices = collections.Counter([np.argmax(conv_output) \\\n","             for conv_output in conv_outputs[filter_size]])\n","        \n","        # Get corresponding text\n","        start = popular_indices.most_common(1)[-1][0]\n","        n_gram = \" \".join([token for token in tokens[start:start+filter_size]])\n","        n_grams[filter_size] = n_gram\n","\n","    return n_grams"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ZN7WVU7BodXR","colab_type":"code","outputId":"765fbf68-5ed3-4925-dc82-dca72e1172ef","executionInfo":{"status":"ok","timestamp":1584478732033,"user_tz":420,"elapsed":2213,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Inputs\n","texts = [\"The Wimbledon tennis tournament starts next week!\",\n","         \"The President signed in the new law.\"]\n","texts = preprocess_texts(texts, lower=LOWER, filters=FILTERS)\n","X_infer = np.array(X_tokenizer.texts_to_sequences(texts))\n","print (f\"{texts[0]} \\n\\t→ {X_tokenizer.sequences_to_texts(X_infer)[0]} \\n\\t→ {X_infer[0]}\")\n","y_filler = np.array([0]*len(texts))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["the wimbledon tennis tournament starts next week \n","\t→ the wimbledon tennis tournament starts next week \n","\t→ [   39 20635   588   622   785   551   576]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"-7HFqMqfok2Q","colab_type":"code","colab":{}},"source":["# Dataset\n","infer_set = TextDataset(X=X_infer, y=y_filler, batch_size=BATCH_SIZE, \n","                        max_filter_size=max(FILTER_SIZES))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"G72dMH1gpRfo","colab_type":"code","colab":{}},"source":["# Iterate over infer batches\n","conv_outputs = collections.defaultdict(list)\n","y_probs = []\n","with torch.no_grad():\n","    for i, (X, y) in enumerate(infer_set.generate_batches()):\n","        \n","        # Set device\n","        X, y = X.to(DEVICE), y.to(DEVICE)\n","\n","        # Forward pass\n","        conv_outputs_, logits = model(X)\n","        y_prob = F.softmax(logits, dim=1)\n","\n","        # Save probabilities\n","        y_probs.extend(y_prob.cpu().numpy())\n","        for i, filter_size in enumerate(FILTER_SIZES):\n","            conv_outputs[filter_size].extend(conv_outputs_[i].cpu().numpy())"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"aXIBmXltpVIi","colab_type":"code","outputId":"6074e98c-2b36-4b57-e5f3-ff17eb087473","executionInfo":{"status":"ok","timestamp":1584478732034,"user_tz":420,"elapsed":2171,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":561}},"source":["# Results\n","results = []\n","for index in range(len(X_infer)):\n","    results.append({\n","        'raw_input': texts[index],\n","        'preprocessed_input': X_tokenizer.sequences_to_texts([X_infer[index]])[0],\n","        'probabilities': get_probability_distribution(y_prob[index], y_tokenizer.classes),\n","        'top_n_grams': get_top_n_grams(\n","            tokens=preprocessed_input.split(' '), \n","            conv_outputs={k:v[index] for k,v in conv_outputs.items()}, \n","            filter_sizes=FILTER_SIZES)})\n","print (json.dumps(results, indent=4))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["[\n","    {\n","        \"raw_input\": \"the wimbledon tennis tournament starts next week\",\n","        \"preprocessed_input\": \"the wimbledon tennis tournament starts next week\",\n","        \"probabilities\": {\n","            \"Sports\": 0.9998615980148315,\n","            \"World\": 0.0001376205327687785,\n","            \"Business\": 7.324182433876558e-07,\n","            \"Sci/Tech\": 7.507998844857866e-08\n","        },\n","        \"top_n_grams\": {\n","            \"2\": \"tournament starts\",\n","            \"3\": \"the wimbledon tennis\",\n","            \"4\": \"tennis tournament starts next\"\n","        }\n","    },\n","    {\n","        \"raw_input\": \"the president signed in the new law\",\n","        \"preprocessed_input\": \"the president signed in the new law\",\n","        \"probabilities\": {\n","            \"World\": 0.6943650245666504,\n","            \"Sports\": 0.14958152174949646,\n","            \"Business\": 0.1257830113172531,\n","            \"Sci/Tech\": 0.03027038462460041\n","        },\n","        \"top_n_grams\": {\n","            \"2\": \"law\",\n","            \"3\": \"the president signed\",\n","            \"4\": \"the president signed in\"\n","        }\n","    }\n","]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kFX3WBAL4Pl8","colab_type":"text"},"source":["Use inferences to collect information how the model performs on your real world data and use it to improve it over time. \n","- Use a probability threshold for the top class (ex. If the predicted class is less than 75%, send the inference for review).\n","- Combine the above with Use probability thresholds for each class (ex. if the predicted class is `Sports` at 85% but that class's precision/recall is low, then send it for review but maybe you don't do this when the predicted class is `Sports` but above 90%.\n","- If the preprocessed sentence has <UNK> tokens, send the inference for further review.\n","- When latency is not an issue, use the n-grams to validate the prediction."]},{"cell_type":"markdown","metadata":{"id":"lMe9PqrxqObm","colab_type":"text"},"source":["Check out the `API` lesson to see how all of this comes together to create an ML service."]},{"cell_type":"markdown","metadata":{"id":"5nWimClvqar7","colab_type":"text"},"source":["---\n","Share and discover ML projects at <a href=\"https://madewithml.com/\">Made With ML</a>.\n","\n","<div align=\"left\">\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/madewithml/lessons\"><img src=\"https://img.shields.io/github/stars/madewithml/lessons.svg?style=social&label=Star\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/madewithml\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/madewithml\"><img src=\"https://img.shields.io/twitter/follow/madewithml.svg?label=Follow&style=social\"></a>\n","</div>\n","             "]}]}